xmap_readers

paddle.fluid.io.xmap_readers(mapper, reader, process_num, buffer_size, order=False)[source]

Use multi-threads to map samples from reader by a mapper defined by user.

Parameters
  • mapper (callable) – A function to map the data from reader.

  • reader (callable) – A data reader which yields the data.

  • process_num (int) – Thread number to handle original sample.

  • buffer_size (int) – Size of the queue to read data in.

  • order (bool) – Whether to keep the data order from original reader. Default False.

Returns

A decorated reader with data mapping.

Example

import paddle.reader as reader
import time

def reader_creator_10(dur):
    def reader():
        for i in range(10):
            time.sleep(dur)
            yield i
    return reader

def mapper(x):
    return (x + 1)

orders = (True, False)
thread_num = (1, 2, 4, 8, 16)
buffer_size = (1, 2, 4, 8, 16)
for order in orders:
    for t_num in thread_num:
        for size in buffer_size:
            user_reader = reader.xmap_readers(mapper,
                                              reader_creator_10(0),
                                              t_num, size, order)
            for n in range(3):
                result = list()
                for i in user_reader():
                    result.append(i)
                if not order:
                    result.sort()
                for idx, e in enumerate(result):
                    assert e == mapper(idx)