DataLoader

class paddle.fluid.io.DataLoader[source]
static from_generator(feed_list=None, capacity=None, use_double_buffer=True, iterable=True, return_list=False)

Create a DataLoader object for loading data from Python generator. Data would be prefetched using Python thread and be pushed into a queue asynchronously.

The created DataLoader object provides 3 methods to set the data source set_sample_generator , set_sample_list_generator and set_batch_generator . Please see the following example codes to know their usages.

If iterable = True, the created DataLoader object is a Python generator object, which is iterable using for-range loop.

If iterable = False, the created DataLoader object provides start() and reset() method to control the data reading process. This mode is designed to be compatible with the fluid.layers.py_reader interface. Users can migrate the codes from fluid.layers.py_reader to fluid.io.DataLoader easily when using iterable=False.

Parameters
  • feed_list (list(Variable)|tuple(Variable)) – feed variable list. The variables should be created by fluid.data().

  • capacity (int) – capacity of the queue maintained in DataLoader. The unit is batch number. Set larger capacity if your reader is fast.

  • use_double_buffer (bool) – whether to use double_buffer_reader. If use_double_buffer=True, the DataLoader would prefetch next batch data asynchronously, so it would speed up data feeding and occupies a little more CPU or GPU memory, i.e., the memory of one batch input data.

  • iterable (bool) – whether the created DataLoader is iterable.

  • return_list (bool) – whether the return value on each device is presented as a list. It is only valid when iterable=True. If return_list=False, the return value on each device would be a dict of str -> LoDTensor, where the key of the dict is the name of each feeded variables. If return_list=True, the return value on each device would be a list(LoDTensor). It is recommended to use return_list=False in static graph mode and use return_list=True in dygraph mode.

Returns

the created DataLoader object.

Return type

loader (DataLoader)

Examples

import paddle.fluid as fluid
import numpy as np

BATCH_NUM = 10
BATCH_SIZE = 16
EPOCH_NUM = 4

CLASS_NUM = 10

ITERABLE = True # whether the created DataLoader object is iterable
USE_GPU = False # whether to use GPU

DATA_FORMAT = 'batch_generator' # data format of data source user provides

def simple_net(image, label):
    fc_tmp = fluid.layers.fc(image, size=CLASS_NUM)
    cross_entropy = fluid.layers.softmax_with_cross_entropy(image, label)
    loss = fluid.layers.reduce_mean(cross_entropy)
    sgd = fluid.optimizer.SGD(learning_rate=1e-3)
    sgd.minimize(loss)
    return loss

def get_random_images_and_labels(image_shape, label_shape):
    image = np.random.random(size=image_shape).astype('float32')
    label = np.random.random(size=label_shape).astype('int64')
    return image, label

# If the data generator yields one sample each time,
# use DataLoader.set_sample_generator to set the data source.
def sample_generator_creator():
    def __reader__():
        for _ in range(BATCH_NUM * BATCH_SIZE):
            image, label = get_random_images_and_labels([784], [1])
            yield image, label

    return __reader__

# If the data generator yield list of samples each time,
# use DataLoader.set_sample_list_generator to set the data source.
def sample_list_generator_creator():
    def __reader__():
        for _ in range(BATCH_NUM):
            sample_list = []
            for _ in range(BATCH_SIZE):
                image, label = get_random_images_and_labels([784], [1])
                sample_list.append([image, label])

            yield sample_list

    return __reader__

# If the data generator yields a batch each time,
# use DataLoader.set_batch_generator to set the data source.
def batch_generator_creator():
    def __reader__():
        for _ in range(BATCH_NUM):
            batch_image, batch_label = get_random_images_and_labels([BATCH_SIZE, 784], [BATCH_SIZE, 1])
            yield batch_image, batch_label

    return __reader__

# If DataLoader is iterable, use for loop to train the network
def train_iterable(exe, prog, loss, loader):
    for _ in range(EPOCH_NUM):
        for data in loader():
            exe.run(prog, feed=data, fetch_list=[loss])

# If DataLoader is not iterable, use start() and reset() method to control the process
def train_non_iterable(exe, prog, loss, loader):
    for _ in range(EPOCH_NUM):
        loader.start() # call DataLoader.start() before each epoch starts
        try:
            while True:
                exe.run(prog, fetch_list=[loss])
        except fluid.core.EOFException:
            loader.reset() # call DataLoader.reset() after catching EOFException

def set_data_source(loader, places):
    if DATA_FORMAT == 'sample_generator':
        loader.set_sample_generator(sample_generator_creator(), batch_size=BATCH_SIZE, drop_last=True, places=places)
    elif DATA_FORMAT == 'sample_list_generator':
        loader.set_sample_list_generator(sample_list_generator_creator(), places=places)
    elif DATA_FORMAT == 'batch_generator':
        loader.set_batch_generator(batch_generator_creator(), places=places)
    else:
        raise ValueError('Unsupported data format')

image = fluid.data(name='image', shape=[None, 784], dtype='float32')
label = fluid.data(name='label', shape=[None, 1], dtype='int64')

# Define DataLoader
loader = fluid.io.DataLoader.from_generator(feed_list=[image, label], capacity=16, iterable=ITERABLE)

# Define network
loss = simple_net(image, label)

# Set data source of DataLoader
#
# If DataLoader is iterable, places must be given and the number of places must be the same with device number.
#  - If you are using GPU, call `fluid.cuda_places()` to get all GPU places.
#  - If you are using CPU, call `fluid.cpu_places()` to get all CPU places.
#
# If DataLoader is not iterable, places can be None.
places = fluid.cuda_places() if USE_GPU else fluid.cpu_places()
set_data_source(loader, places)

exe = fluid.Executor(places[0])
exe.run(fluid.default_startup_program())

prog = fluid.CompiledProgram(fluid.default_main_program()).with_data_parallel(loss_name=loss.name)

if loader.iterable:
    train_iterable(exe, prog, loss, loader)
else:
    train_non_iterable(exe, prog, loss, loader)


'''
Users can use return_list = True in dygraph mode.
'''
with fluid.dygraph.guard(places[0]):
    loader = fluid.io.DataLoader.from_generator(capacity=2, return_list=True)
    set_data_source(loader, places[0])
    for image, label in loader():
        relu = fluid.layers.relu(image)
        assert image.shape == [BATCH_SIZE, 784]
        assert label.shape == [BATCH_SIZE, 1]
        assert relu.shape == [BATCH_SIZE, 784]
static from_dataset(dataset, places, drop_last=True)

Create an iterable DataLoader object for loading data from Dataset. Dataset is only supported in Linux system currently.

Parameters
  • dataset (InMemoryDataset|QueueDataset) – the dataset object.

  • places (list(CUDAPlace)|list(CPUPlace)) – places where the result data should be converted.

  • drop_last (bool) – whether to drop the last batch whose sample number is less than batch size. If drop_last = True, they would be dropped. If drop_last = False, they would be kept.

Returns

the created DataLoader object, which can be

treated as a Python generator.

Return type

loader (DataLoader)

Examples

import paddle.fluid as fluid

image = fluid.data(name='image', shape=[None, 784], dtype='float32')
label = fluid.data(name='label', shape=[None, 1], dtype='int64')

dataset = fluid.DatasetFactory().create_dataset("QueueDataset")
dataset.set_batch_size(32)
dataset.set_filelist(['a.txt', 'b.txt', 'c.txt'])
dataset.set_use_var([image, label])
dataset.set_pipe_command('cat')

loader = fluid.io.DataLoader.from_dataset(dataset, fluid.cpu_places())