layer_norm

api_attr

declarative programming (static graph)

paddle.fluid.layers.layer_norm(input, scale=True, shift=True, begin_norm_axis=1, epsilon=1e-05, param_attr=None, bias_attr=None, act=None, name=None)[source]

Layer Normalization Layer

The API implements the function of the Layer Normalization Layer and can be applied to mini-batch input data. Refer to Layer Normalization

The formula is as follows:

\[ \begin{align}\begin{aligned}\mu & = \frac{1}{H}\sum_{i=1}^{H} x_i\\\sigma & = \sqrt{\frac{1}{H}\sum_{i=1}^{H}{(x_i - \mu)^2} + \epsilon}\\y & = f(\frac{g}{\sigma}(x - \mu) + b)\end{aligned}\end{align} \]
  • \(x\): the vector representation of the summed inputs to the neurons in that layer.

  • \(H\): the number of hidden units in a layers

  • \(\epsilon\): the small value added to the variance to prevent division by zero.

  • \(g\): the trainable scale parameter.

  • \(b\): the trainable bias parameter.

Parameters
  • input (Variable) – A multi-dimension Tensor , and the data type is float32 or float64.

  • scale (bool, optional) – Whether to learn the adaptive gain \(g\) after normalization. Default: True.

  • shift (bool, optional) – Whether to learn the adaptive bias \(b\) after normalization. Default: True.

  • begin_norm_axis (int, optional) – The normalization will be performed along dimensions from begin_norm_axis to rank(input). Default: 1.

  • epsilon (float, optional) – The small value added to the variance to prevent division by zero. Default: 1e-05.

  • param_attr (ParamAttr, optional) – The parameter attribute for the learnable gain \(g\). If scale is False, param_attr is omitted. If scale is True and param_attr is None, a default ParamAttr would be added as scale. The param_attr is initialized as 1 if it is added. Default: None.

  • bias_attr (ParamAttr, optional) – The parameter attribute for the learnable bias \(b\). If shift is False, bias_attr is omitted. If shift is True and param_attr is None, a default ParamAttr would be added as bias. The bias_attr is initialized as 0 if it is added. Default: None.

  • act (str, optional) – Activation to be applied to the output of layer normalization. Default: None.

  • name (str) – The default value is None. Normally there is no need for user to set this property. For more information, please refer to Name .

Returns

Tensor indicating the normalized result, the data type is the same as input , and the return dimension is the same as input .

Return type

Variable

Examples

import paddle.fluid as fluid
import numpy as np
x = fluid.data(name='x', shape=[-1, 32, 32], dtype='float32')
hidden1 = fluid.layers.layer_norm(input=x, begin_norm_axis=1)
place = fluid.CPUPlace()
exe = fluid.Executor(place)
exe.run(fluid.default_startup_program())
np_x = np.random.random(size=(8, 3, 32, 32)).astype('float32')
output = exe.run(feed={"x": np_x}, fetch_list = [hidden1])
print(output)