amp

class paddle.cuda. amp [source]

Namespace for amp marker operations.

static autocast ( enabled=True, dtype=<DataType.FLOAT16: 15>, cache_enabled=True ) → AbstractContextManager

autocast¶

Create a context which enables auto-mixed-precision(AMP) of operators executed in dynamic graph mode. If enabled, the input data type (float32, float16 or bfloat16) of each operator is decided by autocast algorithm for better performance.

Commonly, it is used together with GradScaler and decorator to achieve Auto-Mixed-Precision in imperative mode.

Parameters

• device_type (str, optional) – Device type. But because the paddle does not distinguish between devices, this parameter does not work.

• enable (bool, optional) – Enable auto-mixed-precision or not. Default is True.

• dtype (str, optional) – Whether to use ‘float16’ or ‘bfloat16’. Default is ‘float16’.

• cache_enabled (bool, optional) – whether to enable cache or not. Default is True. But this parameter is not used

Note

paddle.cuda.amp.

Examples

>>> 
>>> import paddle

>>> conv2d = paddle.nn.Conv2D(3, 2, 3, bias_attr=False)
>>> data = paddle.rand([10, 3, 32, 32])

>>> with paddle.device.amp.autocast():
...     conv = conv2d(data)
...     print(conv.dtype)
>>> 
paddle.float16
>>> 

>>> with paddle.device.amp.autocast(enabled=False):
...     conv = conv2d(data)
...     print(conv.dtype)
>>> 
paddle.float32
>>> 

class GradScaler ( enable: bool = True, init_loss_scaling: float = 2.0**16, incr_ratio: float = 2.0, decr_ratio: float = 0.5, incr_every_n_steps: int = 2000, decr_every_n_nan_or_inf: int = 1, use_dynamic_loss_scaling: bool = True )

class GradScaler ( device: str, init_scale: float = 2.0**16, growth_factor: float = 2.0, backoff_factor: float = 0.5, growth_interval: int = 2000, enabled: bool = True )

class GradScaler ( init_scale: float = 2.0**16, growth_factor: float = 2.0, backoff_factor: float = 0.5, growth_interval: int = 2000, enabled: bool = True )

GradScaler is used for Auto-Mixed-Precision training in dynamic graph mode. It controls the scaling of loss, helps avoiding numerical overflow. The object of this class has nineteen methods scale(), unscale_(), minimize(), step(), update() and get/set api of parameters.

scale() is used to multiply the loss by a scale ratio. unscale_() is used to unscale the gradients of parameters, multiplies the gradients of parameters by 1/(scale ratio) minimize() is similar as optimizer.minimize(), performs parameters updating, and it will update the loss_scaling, it equal to step() + update(). step() is similar as optimizer.step(), which performs parameters updating. update is used to update the loss_scaling.

Commonly, it is used together with paddle.amp.auto_cast to achieve Auto-Mixed-Precision in dynamic graph mode.

This API supports three calling conventions:

GradScaler(enable=True, init_loss_scaling=2.0**16, incr_ratio=2.0, decr_ratio=0.5, incr_every_n_steps=2000, decr_every_n_nan_or_inf=1, use_dynamic_loss_scaling=True)

GradScaler(device, init_scale=2.0**16, growth_factor=2.0, backoff_factor=0.5, growth_interval=2000, enabled=True)

GradScaler(init_scale=2.0**16, growth_factor=2.0, backoff_factor=0.5, growth_interval=2000, enabled=True)

Parameters

• enable (bool, optional) – Enable loss scaling or not. Default is True. Alias: enabled

• init_loss_scaling (float, optional) – The initial loss scaling factor. Default is 2.0**16. Alias: init_scale

• incr_ratio (float, optional) –

  The multiplier to use when increasing the loss

  scaling. Default is 2.0.

  System Message: WARNING/2 (/usr/local/lib/python3.10/site-packages/paddle/device/__init__.py:docstring of paddle.device._AutocastMode.autocast, line 18)

  Definition list ends without a blank line; unexpected unindent.

  Alias: growth_factor

• decr_ratio (float, optional) –

  The less-than-one-multiplier to use when decreasing

  the loss scaling. Default is 0.5.

  System Message: WARNING/2 (/usr/local/lib/python3.10/site-packages/paddle/device/__init__.py:docstring of paddle.device._AutocastMode.autocast, line 23)

  Definition list ends without a blank line; unexpected unindent.

  Alias: backoff_factor

• incr_every_n_steps (int, optional) –

  Increases loss scaling every n consecutive

  steps with finite gradients. Default is 2000.

  System Message: WARNING/2 (/usr/local/lib/python3.10/site-packages/paddle/device/__init__.py:docstring of paddle.device._AutocastMode.autocast, line 28)

  Definition list ends without a blank line; unexpected unindent.

  Alias: growth_interval

• decr_every_n_nan_or_inf (int, optional) – Decreases loss scaling every n accumulated steps with nan or inf gradients. Default is 1.

• use_dynamic_loss_scaling (bool, optional) – Whether to use dynamic loss scaling. If False, fixed loss_scaling is used. If True, the loss scaling is updated dynamically. Default is True.

Returns

An GradScaler object.

Examples

>>> import paddle

>>> model = paddle.nn.Conv2D(3, 2, 3, bias_attr=True)
>>> optimizer = paddle.optimizer.SGD(
...     learning_rate=0.01,
...     parameters=model.parameters(),
... )
>>> scaler = paddle.amp.GradScaler(init_loss_scaling=1024)
>>> data = paddle.rand([10, 3, 32, 32])

>>> with paddle.amp.auto_cast():
...     conv = model(data)
...     loss = paddle.mean(conv)

>>> scaled = scaler.scale(loss)  # scale the loss
>>> scaled.backward()  # do backward
>>> scaler.minimize(optimizer, scaled)  # update parameters
>>> optimizer.clear_grad()

get_backoff_factor ( ) → float

get_backoff_factor¶

Get the less-than-one-multiplier to use when decreasing the loss scaling.

Returns

the less-than-one-multiplier to use when decreasing the loss scaling.

Return type

float

Examples

>>> 
>>> import paddle
>>> scaler = paddle.amp.GradScaler(
...     enable=True,
...     init_loss_scaling=1024,
...     incr_ratio=2.0,
...     decr_ratio=0.5,
...     incr_every_n_steps=1000,
...     decr_every_n_nan_or_inf=2,
...     use_dynamic_loss_scaling=True,
... )
>>> decr_ratio = scaler.get_decr_ratio()
>>> print(decr_ratio)
0.5

get_decr_every_n_nan_or_inf ( ) → int

get_decr_every_n_nan_or_inf¶

Return the num n, n represent decreases loss scaling every n accumulated steps with nan or inf gradients.

Returns

the num n, n represent decreases loss scaling every n accumulated steps with nan or inf gradients.

Return type

int

Examples

>>> 
>>> import paddle
>>> scaler = paddle.amp.GradScaler(
...     enable=True,
...     init_loss_scaling=1024,
...     incr_ratio=2.0,
...     decr_ratio=0.5,
...     incr_every_n_steps=1000,
...     decr_every_n_nan_or_inf=2,
...     use_dynamic_loss_scaling=True,
... )
>>> decr_every_n_nan_or_inf = scaler.get_decr_every_n_nan_or_inf()
>>> print(decr_every_n_nan_or_inf)
2

get_decr_ratio ( ) → float

get_decr_ratio¶

Get the less-than-one-multiplier to use when decreasing the loss scaling.

Returns

the less-than-one-multiplier to use when decreasing the loss scaling.

Return type

float

Examples

>>> 
>>> import paddle
>>> scaler = paddle.amp.GradScaler(
...     enable=True,
...     init_loss_scaling=1024,
...     incr_ratio=2.0,
...     decr_ratio=0.5,
...     incr_every_n_steps=1000,
...     decr_every_n_nan_or_inf=2,
...     use_dynamic_loss_scaling=True,
... )
>>> decr_ratio = scaler.get_decr_ratio()
>>> print(decr_ratio)
0.5

get_growth_factor ( ) → float

get_growth_factor¶

Return the multiplier to use when increasing the loss scaling.

Returns

the multiplier to use when increasing the loss scaling.

Return type

float

Examples

>>> 
>>> import paddle
>>> scaler = paddle.amp.GradScaler(
...     enable=True,
...     init_loss_scaling=1024,
...     incr_ratio=2.0,
...     decr_ratio=0.5,
...     incr_every_n_steps=1000,
...     decr_every_n_nan_or_inf=2,
...     use_dynamic_loss_scaling=True,
... )
>>> incr_ratio = scaler.get_incr_ratio()
>>> print(incr_ratio)
2.0

get_growth_interval ( ) → int

get_growth_interval¶

Return the num n, n represent increases loss scaling every n consecutive steps with finite gradients.

Returns

the num n, n represent increases loss scaling every n consecutive steps with finite gradients.

Return type

int

Examples

>>> 
>>> import paddle
>>> scaler = paddle.amp.GradScaler(
...     enable=True,
...     init_loss_scaling=1024,
...     incr_ratio=2.0,
...     decr_ratio=0.5,
...     incr_every_n_steps=1000,
...     decr_every_n_nan_or_inf=2,
...     use_dynamic_loss_scaling=True,
... )
>>> incr_every_n_steps = scaler.get_incr_every_n_steps()
>>> print(incr_every_n_steps)
1000

get_incr_every_n_steps ( ) → int

get_incr_every_n_steps¶

Return the num n, n represent increases loss scaling every n consecutive steps with finite gradients.

Returns

the num n, n represent increases loss scaling every n consecutive steps with finite gradients.

Return type

int

Examples

>>> 
>>> import paddle
>>> scaler = paddle.amp.GradScaler(
...     enable=True,
...     init_loss_scaling=1024,
...     incr_ratio=2.0,
...     decr_ratio=0.5,
...     incr_every_n_steps=1000,
...     decr_every_n_nan_or_inf=2,
...     use_dynamic_loss_scaling=True,
... )
>>> incr_every_n_steps = scaler.get_incr_every_n_steps()
>>> print(incr_every_n_steps)
1000

get_incr_ratio ( ) → float

get_incr_ratio¶

Return the multiplier to use when increasing the loss scaling.

Returns

the multiplier to use when increasing the loss scaling.

Return type

float

Examples

>>> 
>>> import paddle
>>> scaler = paddle.amp.GradScaler(
...     enable=True,
...     init_loss_scaling=1024,
...     incr_ratio=2.0,
...     decr_ratio=0.5,
...     incr_every_n_steps=1000,
...     decr_every_n_nan_or_inf=2,
...     use_dynamic_loss_scaling=True,
... )
>>> incr_ratio = scaler.get_incr_ratio()
>>> print(incr_ratio)
2.0

get_init_loss_scaling ( ) → float

get_init_loss_scaling¶

Return the initial loss scaling factor.

Returns

the initial loss scaling factor.

Return type

float

Examples

>>> 
>>> import paddle
>>> scaler = paddle.amp.GradScaler(
...     enable=True,
...     init_loss_scaling=1024,
...     incr_ratio=2.0,
...     decr_ratio=0.5,
...     incr_every_n_steps=1000,
...     decr_every_n_nan_or_inf=2,
...     use_dynamic_loss_scaling=True,
... )
>>> init_loss_scaling = scaler.get_init_loss_scaling()
>>> print(init_loss_scaling)
1024

get_scale ( ) → float

get_scale¶

Return the current scale factor as a Python float.

If loss scaling is not enabled, returns 0.0.

Returns

The current loss scaling factor, or 0.0 if disabled.

Return type

float

Examples

>>> 
>>> import paddle
>>> scaler = paddle.amp.GradScaler(init_loss_scaling=1024)
>>> print(scaler.get_scale())
1024.0

is_enable ( ) → bool

is_enable¶

Enable loss scaling or not.

Returns

enable loss scaling return True else return False.

Return type

bool

Examples

>>> 
>>> import paddle
>>> scaler = paddle.amp.GradScaler(
...     enable=True,
...     init_loss_scaling=1024,
...     incr_ratio=2.0,
...     decr_ratio=0.5,
...     incr_every_n_steps=1000,
...     decr_every_n_nan_or_inf=2,
...     use_dynamic_loss_scaling=True,
... )
>>> enable = scaler.is_enable()
>>> print(enable)
True

is_enabled ( ) → bool

is_enabled¶

Enable loss scaling or not.

Returns

enable loss scaling return True else return False.

Return type

bool

Examples

>>> 
>>> import paddle
>>> scaler = paddle.amp.GradScaler(
...     enable=True,
...     init_loss_scaling=1024,
...     incr_ratio=2.0,
...     decr_ratio=0.5,
...     incr_every_n_steps=1000,
...     decr_every_n_nan_or_inf=2,
...     use_dynamic_loss_scaling=True,
... )
>>> enable = scaler.is_enable()
>>> print(enable)
True

is_use_dynamic_loss_scaling ( ) → bool

is_use_dynamic_loss_scaling¶

Whether to use dynamic loss scaling.

Returns

if fixed loss_scaling is used return False, if the loss scaling is updated dynamically return True.

Return type

bool

Examples

>>> 
>>> import paddle
>>> scaler = paddle.amp.GradScaler(
...     enable=True,
...     init_loss_scaling=1024,
...     incr_ratio=2.0,
...     decr_ratio=0.5,
...     incr_every_n_steps=1000,
...     decr_every_n_nan_or_inf=2,
...     use_dynamic_loss_scaling=True,
... )
>>> use_dynamic_loss_scaling = scaler.is_use_dynamic_loss_scaling()
>>> print(use_dynamic_loss_scaling)
True

load_state_dict ( state_dict: _ScaleStateDict ) → None

load_state_dict¶

Loads the scaler state.

Parameters

state_dict (dict) – scaler state. Should be an object returned from a call to GradScaler.state_dict().

Examples

>>> 
>>> import paddle

>>> scaler = paddle.amp.GradScaler(
...     enable=True,
...     init_loss_scaling=1024,
...     incr_ratio=2.0,
...     decr_ratio=0.5,
...     incr_every_n_steps=1000,
...     decr_every_n_nan_or_inf=2,
...     use_dynamic_loss_scaling=True,
... )
>>> scaler_state = scaler.state_dict()
>>> scaler.load_state_dict(scaler_state)

minimize ( optimizer: Optimizer | OptimizerWithMixedPrecision, *args: Any, **kwargs: Any ) → tuple[list[Operator], list[tuple[Tensor, Tensor]]]

minimize¶

This function is similar as optimizer.minimize(), which performs parameters updating.

If the scaled gradients of parameters contains NAN or INF, the parameters updating is skipped. Otherwise, if unscale_() has not been called, it first unscales the scaled gradients of parameters, then updates the parameters.

Finally, the loss scaling ratio is updated.

Parameters

• optimizer (Optimizer) – The optimizer used to update parameters.

• args – Arguments, which will be forward to optimizer.minimize().

• kwargs – Keyword arguments, which will be forward to optimizer.minimize().

Examples

>>> import paddle
>>> model = paddle.nn.Conv2D(3, 2, 3, bias_attr=True)
>>> optimizer = paddle.optimizer.SGD(
...     learning_rate=0.01,
...     parameters=model.parameters(),
... )
>>> scaler = paddle.amp.GradScaler(init_loss_scaling=1024)
>>> data = paddle.rand([10, 3, 32, 32])
>>> with paddle.amp.auto_cast():
...     conv = model(data)
...     loss = paddle.mean(conv)
>>> scaled = scaler.scale(loss)  # scale the loss
>>> scaled.backward()  # do backward
>>> scaler.minimize(optimizer, scaled)  # update parameters
>>> optimizer.clear_grad()

scale ( var: Tensor ) → Tensor

scale¶

Multiplies a Tensor by the scale factor and returns scaled outputs. If this instance of GradScaler is not enabled, output are returned unmodified.

Parameters

var (Tensor) – The tensor to scale.

Returns

The scaled tensor or original tensor.

Examples

>>> import paddle
>>> model = paddle.nn.Conv2D(3, 2, 3, bias_attr=True)
>>> optimizer = paddle.optimizer.SGD(
...     learning_rate=0.01,
...     parameters=model.parameters(),
... )
>>> scaler = paddle.amp.GradScaler(init_loss_scaling=1024)
>>> data = paddle.rand([10, 3, 32, 32])
>>> with paddle.amp.auto_cast():
...     conv = model(data)
...     loss = paddle.mean(conv)
>>> scaled = scaler.scale(loss)  # scale the loss
>>> scaled.backward()  # do backward
>>> scaler.minimize(optimizer, scaled)  # update parameters
>>> optimizer.clear_grad()

set_backoff_factor ( new_decr_ratio: float ) → None

set_backoff_factor¶

Set the less-than-one-multiplier to use when decreasing the loss scaling by new_incr_ratio, new_decr_ratio should < 1.0.

Parameters

new_decr_ratio (float) – The new_decr_ratio used to update the less-than-one-multiplier to use when decreasing the loss scaling.

Examples

>>> 
>>> import paddle
>>> scaler = paddle.amp.GradScaler(
...     enable=True,
...     init_loss_scaling=1024,
...     incr_ratio=2.0,
...     decr_ratio=0.5,
...     incr_every_n_steps=1000,
...     decr_every_n_nan_or_inf=2,
...     use_dynamic_loss_scaling=True,
... )
>>> print(scaler.get_decr_ratio())
0.5
>>> new_decr_ratio = 0.1
>>> scaler.set_decr_ratio(new_decr_ratio)
>>> print(scaler.get_decr_ratio())
0.1

set_decr_every_n_nan_or_inf ( new_decr_every_n_nan_or_inf: int ) → None

set_decr_every_n_nan_or_inf¶

Set the num n by new_decr_every_n_nan_or_inf, n represent decreases loss scaling every n accumulated steps with nan or inf gradients.

Parameters

new_decr_every_n_nan_or_inf (int) – The new_decr_every_n_nan_or_inf used to update the num n, n represent decreases loss scaling every n accumulated steps with nan or inf gradients.

Examples

>>> 
>>> import paddle
>>> scaler = paddle.amp.GradScaler(
...     enable=True,
...     init_loss_scaling=1024,
...     incr_ratio=2.0,
...     decr_ratio=0.5,
...     incr_every_n_steps=1000,
...     decr_every_n_nan_or_inf=2,
...     use_dynamic_loss_scaling=True,
... )
>>> print(scaler.get_decr_every_n_nan_or_inf())
2
>>> new_decr_every_n_nan_or_inf = 3
>>> scaler.set_decr_every_n_nan_or_inf(new_decr_every_n_nan_or_inf)
>>> print(scaler.get_decr_every_n_nan_or_inf())
3

set_decr_ratio ( new_decr_ratio: float ) → None

set_decr_ratio¶

Set the less-than-one-multiplier to use when decreasing the loss scaling by new_incr_ratio, new_decr_ratio should < 1.0.

Parameters

new_decr_ratio (float) – The new_decr_ratio used to update the less-than-one-multiplier to use when decreasing the loss scaling.

Examples

>>> 
>>> import paddle
>>> scaler = paddle.amp.GradScaler(
...     enable=True,
...     init_loss_scaling=1024,
...     incr_ratio=2.0,
...     decr_ratio=0.5,
...     incr_every_n_steps=1000,
...     decr_every_n_nan_or_inf=2,
...     use_dynamic_loss_scaling=True,
... )
>>> print(scaler.get_decr_ratio())
0.5
>>> new_decr_ratio = 0.1
>>> scaler.set_decr_ratio(new_decr_ratio)
>>> print(scaler.get_decr_ratio())
0.1

set_growth_factor ( new_incr_ratio: float ) → None

set_growth_factor¶

Set the multiplier to use when increasing the loss scaling by new_incr_ratio, new_incr_ratio should > 1.0.

Parameters

new_incr_ratio (float) – The new_incr_ratio used to update the multiplier to use when increasing the loss scaling.

Examples

>>> 
>>> import paddle
>>> scaler = paddle.amp.GradScaler(
...     enable=True,
...     init_loss_scaling=1024,
...     incr_ratio=2.0,
...     decr_ratio=0.5,
...     incr_every_n_steps=1000,
...     decr_every_n_nan_or_inf=2,
...     use_dynamic_loss_scaling=True,
... )
>>> print(scaler.get_incr_ratio())
2.0
>>> new_incr_ratio = 3.0
>>> scaler.set_incr_ratio(new_incr_ratio)
>>> print(scaler.get_incr_ratio())
3.0

set_growth_interval ( new_incr_every_n_steps: int ) → None

set_growth_interval¶

Set the num n by new_incr_every_n_steps, n represent increases loss scaling every n consecutive steps with finite gradients.

Parameters

new_incr_every_n_steps (int) – The new_incr_every_n_steps used to update the num n, n represent increases loss scaling every n consecutive steps with finite gradients.

Examples

>>> 
>>> import paddle
>>> scaler = paddle.amp.GradScaler(
...     enable=True,
...     init_loss_scaling=1024,
...     incr_ratio=2.0,
...     decr_ratio=0.5,
...     incr_every_n_steps=1000,
...     decr_every_n_nan_or_inf=2,
...     use_dynamic_loss_scaling=True,
... )
>>> print(scaler.get_incr_every_n_steps())
1000
>>> new_incr_every_n_steps = 2000
>>> scaler.set_incr_every_n_steps(new_incr_every_n_steps)
>>> print(scaler.get_incr_every_n_steps())
2000

set_incr_every_n_steps ( new_incr_every_n_steps: int ) → None

set_incr_every_n_steps¶

Set the num n by new_incr_every_n_steps, n represent increases loss scaling every n consecutive steps with finite gradients.

Parameters

new_incr_every_n_steps (int) – The new_incr_every_n_steps used to update the num n, n represent increases loss scaling every n consecutive steps with finite gradients.

Examples

>>> 
>>> import paddle
>>> scaler = paddle.amp.GradScaler(
...     enable=True,
...     init_loss_scaling=1024,
...     incr_ratio=2.0,
...     decr_ratio=0.5,
...     incr_every_n_steps=1000,
...     decr_every_n_nan_or_inf=2,
...     use_dynamic_loss_scaling=True,
... )
>>> print(scaler.get_incr_every_n_steps())
1000
>>> new_incr_every_n_steps = 2000
>>> scaler.set_incr_every_n_steps(new_incr_every_n_steps)
>>> print(scaler.get_incr_every_n_steps())
2000

set_incr_ratio ( new_incr_ratio: float ) → None

set_incr_ratio¶

Set the multiplier to use when increasing the loss scaling by new_incr_ratio, new_incr_ratio should > 1.0.

Parameters

new_incr_ratio (float) – The new_incr_ratio used to update the multiplier to use when increasing the loss scaling.

Examples

>>> 
>>> import paddle
>>> scaler = paddle.amp.GradScaler(
...     enable=True,
...     init_loss_scaling=1024,
...     incr_ratio=2.0,
...     decr_ratio=0.5,
...     incr_every_n_steps=1000,
...     decr_every_n_nan_or_inf=2,
...     use_dynamic_loss_scaling=True,
... )
>>> print(scaler.get_incr_ratio())
2.0
>>> new_incr_ratio = 3.0
>>> scaler.set_incr_ratio(new_incr_ratio)
>>> print(scaler.get_incr_ratio())
3.0

set_init_loss_scaling ( new_init_loss_scaling: int ) → None

set_init_loss_scaling¶

Set the initial loss scaling factor by new_init_loss_scaling.

Parameters

new_init_loss_scaling (float) – The new_init_loss_scaling used to update initial loss scaling factor.

Examples

>>> 
>>> import paddle
>>> scaler = paddle.amp.GradScaler(
...     enable=True,
...     init_loss_scaling=1024,
...     incr_ratio=2.0,
...     decr_ratio=0.5,
...     incr_every_n_steps=1000,
...     decr_every_n_nan_or_inf=2,
...     use_dynamic_loss_scaling=True,
... )
>>> print(scaler.get_init_loss_scaling())
1024
>>> new_init_loss_scaling = 1000
>>> scaler.set_init_loss_scaling(new_init_loss_scaling)
>>> print(scaler.get_init_loss_scaling())
1000

state_dict ( ) → _ScaleStateDict

state_dict¶

Returns the state of the scaler as a dict, If this instance is not enabled, returns an empty dict.

Returns

scale (tensor): The loss scaling factor. incr_ratio(float): The multiplier to use when increasing the loss scaling. decr_ratio(float): The less-than-one-multiplier to use when decreasing the loss scaling. incr_every_n_steps(int): Increases loss scaling every n consecutive steps with finite gradients. decr_every_n_nan_or_inf(int): Decreases loss scaling every n accumulated steps with nan or inf gradients. incr_count(int): The number of recent consecutive unskipped steps. decr_count(int): The number of recent consecutive skipped steps. use_dynamic_loss_scaling(bool): Whether to use dynamic loss scaling. If False, fixed loss_scaling is used. If True, the loss scaling is updated dynamically. Default is True.

Return type

A dict of scaler includes

Examples

>>> 
>>> import paddle

>>> scaler = paddle.amp.GradScaler(
...     enable=True,
...     init_loss_scaling=1024,
...     incr_ratio=2.0,
...     decr_ratio=0.5,
...     incr_every_n_steps=1000,
...     decr_every_n_nan_or_inf=2,
...     use_dynamic_loss_scaling=True,
... )
>>> scaler_state = scaler.state_dict()

step ( optimizer: Optimizer ) → None

step¶

This function is similar as optimizer.step(), which performs parameters updating.

If the scaled gradients of parameters contains NAN or INF, the parameters updating is skipped. Otherwise, if unscale_() has not been called, it first unscales the scaled gradients of parameters, then updates the parameters.

Parameters

optimizer (Optimizer) – The optimizer used to update parameters.

Examples

>>> 
>>> import paddle
>>> paddle.device.set_device('gpu')
>>> model = paddle.nn.Conv2D(3, 2, 3, bias_attr=True)
>>> optimizer = paddle.optimizer.SGD(
...     learning_rate=0.01,
...     parameters=model.parameters(),
... )
>>> scaler = paddle.amp.GradScaler(init_loss_scaling=1024)
>>> data = paddle.rand([10, 3, 32, 32])
>>> with paddle.amp.auto_cast():
...     conv = model(data)
...     loss = paddle.mean(conv)
>>> scaled = scaler.scale(loss)  # scale the loss
>>> scaled.backward()  # do backward
>>> scaler.step(optimizer)  # update parameters
>>> scaler.update()  # update the loss scaling ratio
>>> optimizer.clear_grad()

unscale_ ( optimizer )

unscale_¶

Unscale the gradients of parameters, multiplies the gradients of parameters by 1/(loss scaling ratio). If this instance of GradScaler is not enabled, output are returned unmodified.

Parameters

optimizer (Optimizer) – The optimizer used to update parameters.

Returns

The unscaled parameters or original parameters.

Examples

>>> 
>>> import paddle
>>> paddle.device.set_device('gpu')
>>> model = paddle.nn.Conv2D(3, 2, 3, bias_attr=True)
>>> optimizer = paddle.optimizer.SGD(
...     learning_rate=0.01,
...     parameters=model.parameters(),
... )
>>> scaler = paddle.amp.GradScaler(init_loss_scaling=1024)
>>> data = paddle.rand([10, 3, 32, 32])
>>> with paddle.amp.auto_cast():
...     conv = model(data)
...     loss = paddle.mean(conv)
>>> scaled = scaler.scale(loss)  # scale the loss
>>> scaled.backward()  # do backward
>>> scaler.unscale_(optimizer)  # unscale the parameter
>>> scaler.step(optimizer)
>>> scaler.update()
>>> optimizer.clear_grad()

update ( new_scale: Optional[float] = None ) → None

update¶

Updates the loss_scaling.

Parameters

new_scale (float, optional) – New loss scaling factor. If provided, the loss scaling factor is directly set to new_scale and the internal step counts are reset. Default is None.

Examples

>>> 
>>> import paddle
>>> paddle.device.set_device('gpu')
>>> model = paddle.nn.Conv2D(3, 2, 3, bias_attr=True)
>>> optimizer = paddle.optimizer.SGD(
...     learning_rate=0.01,
...     parameters=model.parameters(),
... )
>>> scaler = paddle.amp.GradScaler(init_loss_scaling=1024)
>>> data = paddle.rand([10, 3, 32, 32])
>>> with paddle.amp.auto_cast():
...     conv = model(data)
...     loss = paddle.mean(conv)
>>> scaled = scaler.scale(loss)  # scale the loss
>>> scaled.backward()  # do backward
>>> scaler.step(optimizer)  # update parameters
>>> scaler.update()  # update the loss scaling ratio
>>> optimizer.clear_grad()