ParallelMode

class paddle.distributed. ParallelMode [source]

There are all the parallel modes currently supported:

  • DATA_PARALLEL: Distribute input data to different devices.

  • TENSOR_PARALLEL: Shards tensors in the network to different devices.

  • PIPELINE_PARALLEL: Place different layers of the network on different devices.

  • SHARDING_PARALLEL: Segment the model parameters, parameter gradients and optimizer states corresponding to the parameters to each device.

Examples

>>> 
>>> import paddle
>>> parallel_mode = paddle.distributed.ParallelMode
>>> print(parallel_mode.DATA_PARALLEL)
0