AmpScaler

class paddle.fluid.dygraph.amp.loss_scaler. AmpScaler ( enable=True, init_loss_scaling=32768.0, incr_ratio=2.0, decr_ratio=0.5, incr_every_n_steps=1000, decr_every_n_nan_or_inf=1, use_dynamic_loss_scaling=True ) [source]
Api_attr

imperative

AmpScaler is used for Auto-Mixed-Precision training/inferring in imperative mode. It controls the scaling of loss, helps avoiding numerical overflow. The object of this class has two methods scale(), minimize().

scale() is used to multiply the loss by a scale ratio. minimize() is similar as Optimizer.minimize(), performs parameters updating.

Commonly, it is used together with amp_guard to achieve Auto-Mixed-Precision in imperative mode.

Parameters
  • enable (bool, optional) – Enable loss scaling or not. Default is True.

  • init_loss_scaling (float, optional) – The initial loss scaling factor. Default is 2**15.

  • incr_ratio (float, optional) – The multiplier to use when increasing the loss scaling. Default is 2.0.

  • decr_ratio (float, optional) – The less-than-one-multiplier to use when decreasing the loss scaling. Default is 0.5.

  • incr_every_n_steps (int, optional) – Increases loss scaling every n consecutive steps with finite gradients. Default is 1000.

  • decr_every_n_nan_or_inf (int, optional) – Decreases loss scaling every n accumulated steps with nan or inf gradients. Default is 2.

  • use_dynamic_loss_scaling (bool, optional) – Whether to use dynamic loss scaling. If False, fixed loss_scaling is used. If True, the loss scaling is updated dynamicly. Default is True.

Returns

An AmpScaler object.

Examples

import numpy as np
import paddle.fluid as fluid

data = np.random.uniform(-1, 1, [10, 3, 32, 32]).astype('float32')
with fluid.dygraph.guard():
    model = fluid.dygraph.Conv2D(3, 2, 3)
    optimizer = fluid.optimizer.SGDOptimizer(
            learning_rate=0.01, parameter_list=model.parameters())
    scaler = fluid.dygraph.AmpScaler(init_loss_scaling=1024)
    data = fluid.dygraph.to_variable(data)
    with fluid.dygraph.amp_guard():
        conv = model(data)
        loss = fluid.layers.reduce_mean(conv)
        scaled = scaler.scale(loss)
        scaled.backward()
        scaler.minimize(optimizer, scaled)
scale ( var )

scale

Multiplies a variable(Tensor) by the scale factor and returns scaled outputs. If this instance of AmpScaler is not enabled, output are returned unmodified.

Parameters

var (Variable) – The variable to scale.

Returns

The scaled variable or original variable.

Examples


import numpy as np import paddle.fluid as fluid

data = np.random.uniform(-1, 1, [10, 3, 32, 32]).astype(‘float32’) with fluid.dygraph.guard():

System Message: ERROR/3 (/usr/local/lib/python3.8/site-packages/paddle/fluid/dygraph/amp/loss_scaler.py:docstring of paddle.fluid.dygraph.amp.loss_scaler.AmpScaler.scale, line 18)

Unexpected indentation.

model = fluid.dygraph.Conv2D(3, 2, 3) optimizer = fluid.optimizer.SGDOptimizer(

System Message: ERROR/3 (/usr/local/lib/python3.8/site-packages/paddle/fluid/dygraph/amp/loss_scaler.py:docstring of paddle.fluid.dygraph.amp.loss_scaler.AmpScaler.scale, line 20)

Unexpected indentation.

learning_rate=0.01, parameter_list=model.parameters())

System Message: WARNING/2 (/usr/local/lib/python3.8/site-packages/paddle/fluid/dygraph/amp/loss_scaler.py:docstring of paddle.fluid.dygraph.amp.loss_scaler.AmpScaler.scale, line 21)

Block quote ends without a blank line; unexpected unindent.

scaler = fluid.dygraph.AmpScaler(init_loss_scaling=1024) data = fluid.dygraph.to_variable(data) with fluid.dygraph.amp_guard():

System Message: ERROR/3 (/usr/local/lib/python3.8/site-packages/paddle/fluid/dygraph/amp/loss_scaler.py:docstring of paddle.fluid.dygraph.amp.loss_scaler.AmpScaler.scale, line 24)

Unexpected indentation.

conv = model(data) loss = fluid.layers.reduce_mean(conv) scaled = scaler.scale(loss) scaled.backward() scaler.minimize(optimizer, scaled)

minimize ( optimizer, *args, **kwargs )

minimize

This function is similar as Optimizer.minimize(), which performs parameters updating.

If the scaled gradients of parameters contains NAN or INF, the parameters updating is skipped. Otherwise, it first unscales the scaled gradients of parameters, then updates the parameters.

Finally, the loss scaling ratio is updated.

Parameters
  • optimizer (Optimizer) – The optimizer used to update parameters.

  • args – Arguments, which will be forward to optimizer.minimize().

  • kwargs – Keyword arguments, which will be forward to Optimizer.minimize().

Examples


import numpy as np import paddle.fluid as fluid

data = np.random.uniform(-1, 1, [10, 3, 32, 32]).astype(‘float32’) with fluid.dygraph.guard():

System Message: ERROR/3 (/usr/local/lib/python3.8/site-packages/paddle/fluid/dygraph/amp/loss_scaler.py:docstring of paddle.fluid.dygraph.amp.loss_scaler.AmpScaler.minimize, line 22)

Unexpected indentation.

model = fluid.dygraph.Conv2D(3, 2, 3) optimizer = fluid.optimizer.SGDOptimizer(

System Message: ERROR/3 (/usr/local/lib/python3.8/site-packages/paddle/fluid/dygraph/amp/loss_scaler.py:docstring of paddle.fluid.dygraph.amp.loss_scaler.AmpScaler.minimize, line 24)

Unexpected indentation.

learning_rate=0.01, parameter_list=model.parameters())

System Message: WARNING/2 (/usr/local/lib/python3.8/site-packages/paddle/fluid/dygraph/amp/loss_scaler.py:docstring of paddle.fluid.dygraph.amp.loss_scaler.AmpScaler.minimize, line 25)

Block quote ends without a blank line; unexpected unindent.

scaler = fluid.dygraph.AmpScaler(init_loss_scaling=1024) data = fluid.dygraph.to_variable(data) with fluid.dygraph.amp_guard():

System Message: ERROR/3 (/usr/local/lib/python3.8/site-packages/paddle/fluid/dygraph/amp/loss_scaler.py:docstring of paddle.fluid.dygraph.amp.loss_scaler.AmpScaler.minimize, line 28)

Unexpected indentation.

conv = model(data) loss = fluid.layers.reduce_mean(conv) scaled = scaler.scale(loss) scaled.backward() scaler.minimize(optimizer, scaled)