MSRAInitializer

class paddle.fluid.initializer. MSRAInitializer ( uniform=True, fan_in=None, seed=0, negative_slope=0, nonlinearity='relu' ) [source]

Implements the MSRA initializer a.k.a. Kaiming Initializer

This class implements the weight initialization from the paper Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification by Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun. This is a robust initialization method that particularly considers the rectifier nonlinearities. In case of Uniform distribution, the range is [-x, x], where

\[x = gain \times \sqrt{\frac{3}{fan\_in}}\]

In case of Normal distribution, the mean is 0 and the standard deviation is

\[\frac{gain}{\sqrt{{fan\_in}}}\]
Parameters

  • uniform (bool, optional) – whether to use uniform or normal distribution

  • fan_in (float32|None, optional) – fan_in (in_features) of trainable Tensor, If None, it will be infered automaticly. If you don’t want to use in_features of the Tensor, you can set the value of ‘fan_in’ smartly by yourself. default is None.

  • seed (int32, optional) – random seed.

  • negative_slope (float, optional) – negative_slope (only used with leaky_relu). default is 0.0.

  • nonlinearity (str, optional) – the non-linear function. default is relu.

Note

It is recommended to set fan_in to None for most cases.

Examples

import paddle
import paddle.fluid as fluid
paddle.enable_static()
x = fluid.data(name="data", shape=[8, 32, 32], dtype="float32")
fc = fluid.layers.fc(input=x, size=10,
    param_attr=fluid.initializer.MSRA(uniform=False))
forward ( var, block=None )

forward

Initialize the input tensor with MSRA initialization.

Parameters

  • var (Tensor) – Tensor that needs to be initialized.

  • block (Block, optional) – The block in which initialization ops should be added. Used in static graph only, default None.

Returns

The initialization op