InverseTimeDecay¶

api_attr

imperative programming (dynamic graph)

class paddle.fluid.dygraph.InverseTimeDecay(learning_rate, decay_steps, decay_rate, staircase=False, begin=0, step=1, dtype='float32')[source]

Applies inverse time decay to the initial learning rate.

The algorithm can be described as following. If staircase is set to False, then:

\[decayed\_learning\_rate = \frac{learning\_rate}{1 + decay\_rate * \frac{global\_step}{decay\_step}}\]

If staircase is set to True, then:

\[decayed\_learning\_rate = \frac{learning\_rate}{1 + decay\_rate * math.floor(\frac{global\_step}{decay\_step})}\]

Parameters

• learning_rate (Variable|float) – The initial learning rate. If the type is Variable, it’s a tensor with shape [1], the data type can be float32 or float64. It also can be set to python int number.

• decay_steps (int) – The decay step size. It determines the decay cycle.

• decay_rate (float) – The decay rate.

• staircase (bool, optional) – If set to True, decay the learning rate at discrete intervals. The default value is False.

• begin (int, optional) – The begin step. The initial value of global_step described above. The default value is 0.

• step (int, optional) – The step size used to calculate the new global_step in the description above. The default value is 1.

• dtype (str, optional) – The data type used to create the learning rate variable. The data type can be ‘float32’, ‘float64’. The default value is ‘float32’.

Returns

None.

Examples

import paddle.fluid as fluid
base_lr = 0.1
with fluid.dygraph.guard():
    emb = fluid.dygraph.Embedding([10, 10])
    sgd_optimizer = fluid.optimizer.SGD(
        learning_rate=fluid.dygraph.InverseTimeDecay(
              learning_rate=base_lr,
              decay_steps=10000,
              decay_rate=0.5,
              staircase=True),
        parameter_list = emb.parameters())