RReLU¶

class paddle.nn. RReLU ( lower=0.125, upper=0.3333333333333333, name=None ) [source]

RReLU activation layer.

Applies the randomized leaky rectified liner unit function to improve generalization performance, as described in the paper: Empirical Evaluation of Rectified Activations in Convolutional Network

During training, randomly samples the negative slope for activation values as described below:

$\begin{split}RReLU(x)= \left\{ \begin{array}{rcl} x, & & if \ x >= 0 \\ a * x, & & otherwise \\ \end{array} \right.\end{split}$

where $$x$$ is the input tensor, $$a$$ is randomly sampled from uniform distribution in range ($$lower$$, $$upper$$),

In the test phase, the negative slope will take the average value of $$lower$$ and $$upper$$:

$\begin{split}RReLU(x)= \left\{ \begin{array}{rcl} x, & & if \ x >= 0 \\ (lower + upper) * 0.5 * x, & & otherwise \\ \end{array} \right.\end{split}$

where $$x$$ is the input tensor, $$lower$$ and $$upper$$ are the bounds of uniform distribution.

Parameters
• lower (float, optional) – The lower bound of uniform distribution. Default: 0.125.

• upper (float, optional) – The upper bound of uniform distribution. Default: 0.333.

• name (str, optional) – Name for the operation (optional, default is None). For more information, please refer to Name.

Shape:
• input: Tensor with any shape. Default dtype is float32.

• output: Tensor with the same shape as input.

Examples

import paddle

input_tensor = paddle.to_tensor([[[[-2.0,  3.0, -4.0,  5.0],
[ 3.0, -4.0,  5.0, -6.0],
[-7.0, -8.0,  8.0,  9.0]],
[[ 1.0, -2.0, -3.0,  4.0],
[-5.0,  6.0,  7.0, -8.0],
[ 6.0,  7.0,  8.0,  9.0]]]], dtype='float32')

out = rrelu_layer(input_tensor)
print(out)
#[[[[-0.20000899  3.         -0.88108218  5.        ]
#   [ 3.         -0.55175185  5.         -1.07761011]
#   [-1.06806871 -1.98962009  8.          9.        ]]
#  [[ 1.         -0.52382672 -0.65515128  4.        ]
#   [-1.37663394  6.          7.         -2.34657836]
#   [ 6.          7.          8.          9.        ]]]]

forward ( x )

Defines the computation performed at every call. Should be overridden by all subclasses.

Parameters
• *inputs (tuple) – unpacked tuple arguments

• **kwargs (dict) – unpacked dict arguments

extra_repr ( )

Extra representation of this layer, you can have custom implementation of your own layer.