Model Parameters¶
Note
The paddle.fluid.* APIs are deprecated. Please use the latest Paddle API versions instead.
Model parameters are usually weights and bias in the model. In Paddle’s dynamic graph mode, they correspond to the EagerParamBase
class. Model parameters are learnable variables that have gradients and can be optimized. In Paddle, custom parameters can be created using api_paddle_create_parameter .
You can configure properties related to model parameters using api_paddle_ParamAttr . The configurable options include:
Initialization method
Regularization
Model averaging
Clipping
Initialization Method¶
Paddle initializes a single parameter by setting attributes of initializer
in ParamAttr
.Example:
import paddle param_attrs = paddle.ParamAttr(name="fc_weight", initializer=paddle.nn.initializer.Constant(5.0)) fc_layer = paddle.nn.Linear(64, 10, weight_attr=param_attrs)
The following is the initialization method supported by Paddle:
1. Constant¶
The constant initialization method sets parameters to a fixed value, such as initializing biases to 0.
Parameter: value specifies the initial value (default is 0.0).
API reference: api_paddle_nn_initializer_Constant
2. Normal¶
The random normal distribution method generates values based on a normal (Gaussian) distribution, suitable for initializing most neural network parameters.
Parameters: mean (default 0.0) and std (default 1.0) define the mean and standard deviation.
API reference: api_paddle_nn_initializer_Normal
3. Uniform¶
The random uniform distribution method samples values evenly within a specified range [low, high].
Parameters: low (default -1.0) and high (default 1.0) define the range.
API reference: api_paddle_nn_initializer_Uniform
4. XavierUniform¶
The Xavier uniform distribution method, proposed by Xavier Glorot and Yoshua Bengio in the paper Understanding the difficulty of training deep feedforward neural networks, initializes parameters based on a uniform distribution.
Range is determined by fan_in (input dimension), fan_out (output dimension), and gain (scaling factor).
API reference: api_paddle_nn_initializer_XavierUniform
5. XavierNormal¶
The Xavier normal distribution method, proposed in the paper Understanding the difficulty of training deep feedforward neural networks, initializes parameters with a normal distribution.
Mean is 0, and the standard deviation is determined by fan_in, fan_out, and gain .
API reference: api_paddle_nn_initializer_XavierNormal
6. KaimingUniform¶
The Kaiming uniform distribution method, proposed by Kaiming He et al. in the paper Delving Deep into Rectifiers, is designed for networks with specific activation functions.
Range is determined by fan_in, negative_slope (default 0), and nonlinearity (default ‘relu’).
API reference: api_paddle_nn_initializer_KaimingUniform
7. KaimingNormal¶
The Kaiming normal distribution method, proposed in the paper Delving Deep into Rectifiers, uses a normal distribution.
Mean is 0, and the standard deviation is determined by fan_in, negative_slope (default 0), and nonlinearity (default ‘relu’).
API reference: api_paddle_nn_initializer_KaimingNormal
8. TruncatedNormal¶
The truncated normal distribution method limits the generated values from a normal distribution to a specified range [a, b].
Parameters: mean (default 0.0), std (default 1.0), and truncation bounds a and b (default -2.0 and 2.0).
API reference: api_paddle_nn_initializer_TruncatedNormal
Other Initialization Methods¶
Paddle also supports the following initialization methods:
api_paddle_nn_initializer_Assign: Initialize directly using a NumPy array, Python list, or Tensor.
api_paddle_nn_initializer_Bilinear: Used for upsampling in transposed convolutions to enlarge feature maps.
api_paddle_nn_initializer_Dirac: Initializes convolution kernels with a Dirac delta function to preserve input characteristics.
api_paddle_nn_initializer_Orthogonal: Generates an orthogonal matrix for initialization, ensuring (semi-)orthogonality.
api_paddle_nn_initializer_set_global_initializer: Sets a global initialization method, effective only for code that follows it.
Regularization¶
Paddle regularizes a single parameter by setting attributes of regularizer
in ParamAttr
.
import paddle param_attrs = paddle.ParamAttr(name="fc_weight", regularizer=paddle.regularizer.L1Decay(0.1)) fc_layer = paddle.nn.Linear(64, 10, weight_attr=param_attrs)
The following is the regularization approach supported by Paddle:
api_paddle_regularizer_L1Decay
api_paddle_regularizer_L2Decay
Model Averaging¶
Paddle determines whether to average a single parameter by setting attributes of do_model_average
in ParamAttr
.
Default value: True .
import paddle param_attrs = paddle.ParamAttr(name="fc_weight", do_model_average=True) fc_layer = paddle.nn.Linear(64, 10, weight_attr=param_attrs)
During mini-batch training, the model parameters are updated after each batch. Model averaging calculates the average of the parameters from the most recent k updates.
The averaged parameters are used only for testing and prediction, not for training.
API reference: api_paddle_incubate_ModelAverage (currently in incubation and may undergo changes).
Clipping¶
Note
The gradient_clip
attribute is deprecated. Use need_clip
to control whether gradient clipping is applied, and configure the clipping method when initializing the optimizer
.
Paddle determines whether gradient clipping is applied to a single parameter by setting attributes of need_clip
in ParamAttr
.
Default value: True .
import paddle param_attrs = paddle.ParamAttr(name="fc_weight", need_clip=True) fc_layer = paddle.nn.Linear(64, 10, weight_attr=param_attrs)
The following is the clipping method supported by Paddle:
1. GradientClipByGlobalNorm¶
Limits the sum of the L2 norms of all Tensors in a Tensor list t_list to the clip_norm
range.
API reference: api_paddle_nn_ClipGradByGlobalNorm
2. GradientClipByNorm¶
Limits the L2 norm of a multi-dimensional input Tensor X to the clip_norm
range.
API reference: api_paddle_nn_ClipGradByNorm
3. GradientClipByValue¶
Limits the values of a multi-dimensional input Tensor X to the range [min, max].
API reference: api_paddle_nn_ClipGradByValue
For more details on gradient clipping methods, refer to: Gradient clip methods in Paddle .