Initialize parallel training environment in dynamic graph mode.
Note
Now initialize both NCCL and GLOO contexts for communication.
Parameters
backend (string) – A string represents the backend used by DataParallel, should be one of ‘gloo’(for cpu), ‘nccl’(for cuda), ‘bkcl’(for xpu), ‘auto’(auto detect). The auto detection prefer ‘nccl’, ‘bkcl’ than ‘gloo’.
Returns
None
Examples
System Message: ERROR/3 (/usr/local/lib/python3.8/site-packages/paddle/distributed/parallel.py:docstring of paddle.distributed.parallel.init_parallel_env, line 15)
Error in “code-block” directive: maximum 1 argument(s) allowed, 18 supplied.
..code-block::python# required: gpuimportpaddleimportpaddle.nnasnnimportpaddle.optimizerasoptimportpaddle.distributedasdistclassLinearNet(nn.Layer):def__init__(self):super(LinearNet,self).__init__()self._linear1=nn.Linear(10,10)self._linear2=nn.Linear(10,1)defforward(self,x):returnself._linear2(self._linear1(x))deftrain():# 1. initialize parallel environmentdist.init_parallel_env()# 2. create data parallel layer & optimizerlayer=LinearNet()dp_layer=paddle.DataParallel(layer)loss_fn=nn.MSELoss()adam=opt.Adam(learning_rate=0.001,parameters=dp_layer.parameters())# 3. run layerinputs=paddle.randn([10,10],'float32')outputs=dp_layer(inputs)labels=paddle.randn([10,1],'float32')loss=loss_fn(outputs,labels)loss.backward()adam.step()adam.clear_grad()if__name__=='__main__':dist.spawn(train)