DistributedOptimizer

class paddle.fluid.incubate.fleet.base.fleet_base. DistributedOptimizer ( optimizer, strategy=None ) [source]

DistributedOptimizer is a wrapper for paddle.fluid.optimizer A user should pass a paddle.fluid.optimizer to DistributedOptimizer minimize() function is implemented. DistributedOptimizer is the starting point for a user who wants to run distributed training. The optimized information will be stored in Fleet() instance who holds the global information about current distributed training.

Parameters

  • optimizer (Optimizer) – subclass of Optimizer.

  • strategy (any) – the user define config for Optimizer.

Returns

None

abstract backward ( loss, startup_program=None, parameter_list=None, no_grad_set=None, callbacks=None )

First part of minimize, do auto-diff to append backward ops for the current program.

Parameters

  • loss (Variable) – loss variable to run optimizations.

  • startup_program (Program) – startup_program for initializing parameters in parameter_list.

  • parameter_list (list) – list of Variables to update.

  • no_grad_set (set|None) – set of Variables should be ignored.

  • callbacks (list|None) – list of callables to run when appending backward operator for one parameter.

Returns

list of (param, grad) pair, grad is the output of backward.

Return type

list

Examples

See examples in apply_gradients.

abstract apply_gradients ( params_grads )

Second part of minimize, appending optimization operators for given params_grads pairs.

Parameters

params_grads (list) – list of (param, grad) pair to do optimization.

Returns

A list of operators appended to the current program.

Return type

list

Examples

loss = network()
optimizer = fluid.optimizer.SGD(learning_rate=0.1)
params_grads = optimizer.backward(loss)
# you may append operations for params_grads here
# ...
optimizer.apply_gradients(params_grads)
abstract minimize ( losses, scopes=None, startup_programs=None, parameter_list=None, no_grad_set=None )

Add operations to minimize loss by updating parameter_list.

This method combines interface backward() and apply_gradients() into one.

Parameters

  • losses (Variable|Variable List) – loss variable to run optimizations.

  • scopes (Scope| Scope List) – scope instance.

  • startup_programs (Program|Program List) – startup_program for initializing parameters in parameter_list.

  • parameter_list (list) – list of Variables to update.

  • no_grad_set (set|None) – set of Variables should be ignored.

Returns

(optimize_ops, params_grads) which are, list of operators appended; and list of (param, grad) Variables pair for optimization.

Return type

tuple