weight_norm¶

paddle.nn.utils. weight_norm ( layer, name='weight', dim=0 ) [source]

This weight_norm layer applies weight normalization to a parameter according to the following formula:

$\mathbf{w} = g \dfrac{v}{\|v\|}$

Weight normalization is a reparameterization of the weight vectors in a neural network that decouples the magnitude of those weight vectors from their direction. Weight normalization replaces the parameter specified by name(eg: ‘weight’) with two parameters: one parameter specifying the magnitude (eg: ‘weight_g’) and one parameter specifying the direction (eg: ‘weight_v’). Weight normalization has been implemented as discussed in this paper: Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks.

Parameters
• layer (Layer) – Layer of paddle, which has weight.

• name (str, optional) – Name of the weight parameter. Default: ‘weight’.

• dim (int, optional) – Dimension over which to compute the norm. Dim is a non-negative number which is less than the rank of weight Tensor. For Example, dim can be chosen from 0, 1, 2, 3 for convolution whose weight shape is [cout, cin, kh, kw] and rank is 4. If dim is set to None, meaning that all elements will be normalized. Default: 0.

Returns

Origin layer with weight norm hook.

Examples

import numpy as np