decorate_bf16¶

paddle.fluid.contrib.mixed_precision.bf16.decorator. decorate_bf16 ( optimizer, amp_lists=None, use_pure_bf16=False, use_bf16_guard=None ) [source]

Decorate the given optimizer to adapt to the mixed-precision training.

Parameters

• optimizer (Optimizer) – A common Optimizer.

• amp_lists (CustomOpLists) – An CustomOpLists object.

• use_pure_bf16 (bool) – Whether to use the pure bf16 training. Default False.

• use_bf16_guard (bool) – Whether to use bf16_guard when constructing the program. Default None, which means that its value equals to use_pure_bf16.

Returns

An optimizer acting like a normal one but with mixed-precision training enabled.

Examples 1:

# fp32&bf16 list based strategy example import paddle import paddle.static as static

paddle.enable_static()

data = static.data(name=’X’, shape=[None, 1], dtype=’float32’) hidden = static.nn.fc(x=data, size=10) loss = paddle.mean(hidden) optimizer = paddle.optimizer.Adam(learning_rate=0.001)

mp_optimizer = static.amp.decorate_bf16(optimizer=optimizer)

ops, param_grads = mp_optimizer.minimize(loss)

Examples 2:

# pure bf16 training example
import numpy as np
import paddle
import paddle.nn.functional as F

def run_example_code():
    place = paddle.CPUPlace(0)
    exe = paddle.static.Executor(place)
    data = paddle.static.data(name='X', shape=[None, 1, 28, 28], dtype='float32')
    conv2d = paddle.static.nn.conv2d(input=data, num_filters=6, filter_size=3)
    # 1) Use bf16_guard to control the range of bf16 kernels used.
    with paddle.static.amp.bf16_guard():
        bn = paddle.static.nn.batch_norm(input=conv2d, act="relu")
        pool = F.max_pool2d(bn, kernel_size=2, stride=2)
        hidden = paddle.static.nn.fc(pool, size=10)
        loss = paddle.mean(hidden)
    # 2) Create the optimizer and set `multi_precision` to True.
    # Setting `multi_precision` to True can avoid the poor accuracy
    # or the slow convergence in a way.
    optimizer = paddle.optimizer.Momentum(learning_rate=0.01, multi_precision=True)
    # 3) These ops in `custom_fp32_list` will keep in the float32 computation type.
    amp_list = paddle.static.amp.CustomOpLists(
        custom_fp32_list=['pool2d'])
    # 4) The entry of Paddle AMP.
    # Enable pure bf16 training by setting `use_pure_bf16` to True.
    optimizer = paddle.static.amp.decorate_bf16(
        optimizer,
        amp_list,
        use_pure_bf16=True)
    # If you don't use the default_startup_program(), you sholud pass
    # your defined `startup_program` into `minimize`.
    optimizer.minimize(loss)
    exe.run(paddle.static.default_startup_program())
    # 5) Use `amp_init` after FP32 parameters initialization(such as `exe.run(startup_program)`).
    # If you want to perform the testing process, you should pass `test_program` into `amp_init`.
    optimizer.amp_init(place, scope=paddle.static.global_scope())