paddle.nn.utils. weight_norm ( layer, name='weight', dim=0 ) [source]

This weight_norm layer applies weight normalization to a parameter according to the following formula:

\[\mathbf{w} = g \dfrac{v}{\|v\|}\]

Weight normalization is a reparameterization of the weight vectors in a neural network that decouples the magnitude of those weight vectors from their direction. Weight normalization replaces the parameter specified by name`(eg: ‘weight’) with two parameters: one parameter specifying the magnitude (eg: ‘weight_g’) and one parameter specifying the direction (eg: ‘weight_v’). Weight normalization has been implemented as discussed in this paper: `Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks.

  • layer (Layer) – Layer of paddle, which has weight.

  • name (str, optional) – Name of the weight parameter. Default: ‘weight’.

  • dim (int, optional) – Dimension over which to compute the norm. Dim is a non-negative number which is less than the rank of weight Tensor. For Example, dim can be chosen from 0, 1, 2, 3 for convolution whose weight shape is [cout, cin, kh, kw] and rank is 4. If dim is set to None, meaning that all elements will be normalized. Default: 0.


Origin layer with weight norm hook.


import numpy as np
from paddle.nn import Conv2D
from paddle.nn.utils import weight_norm

x = np.array([[[[0.3, 0.4], [0.3, 0.07]], [[0.83, 0.37], [0.18, 0.93]]]]).astype('float32')
conv = Conv2D(3, 5, 3)
wn = weight_norm(conv)
# [5]
# [5, 3, 3, 3]