dynamic_lstm

api_attr

declarative programming (static graph)

paddle.fluid.layers.dynamic_lstm(input, size, h_0=None, c_0=None, param_attr=None, bias_attr=None, use_peepholes=True, is_reverse=False, gate_activation='sigmoid', cell_activation='tanh', candidate_activation='tanh', dtype='float32', name=None)[source]
Note:
  1. This OP only supports LoDTensor as inputs. If you need to deal with Tensor, please use lstm .

  2. In order to improve efficiency, users must first map the input of dimension [T, hidden_size] to input of [T, 4 * hidden_size], and then pass it to this OP.

The implementation of this OP include diagonal/peephole connections. Please refer to Gers, F. A., & Schmidhuber, J. (2000) . If you do not need peephole connections, please set use_peepholes to False .

This OP computes each timestep as follows:

\[i_t = \sigma(W_{ix}x_{t} + W_{ih}h_{t-1} + b_{x_i} + b_{h_i})\]
\[f_t = \sigma(W_{fx}x_{t} + W_{fh}h_{t-1} + b_{x_f} + b_{h_f})\]
\[o_t = \sigma(W_{ox}x_{t} + W_{oh}h_{t-1} + b_{x_o} + b_{h_o})\]
\[\widetilde{c_t} = tanh(W_{cx}x_t + W_{ch}h_{t-1} + b{x_c} + b_{h_c})\]
\[c_t = f_t \odot c_{t-1} + i_t \odot \widetilde{c_t}\]
\[h_t = o_t \odot tanh(c_t)\]

The symbolic meanings in the formula are as follows:

  • \(x_{t}\) represents the input at timestep \(t\)

  • \(h_{t}\) represents the hidden state at timestep \(t\)

  • \(h_{t-1}, c_{t-1}\) represent the hidden state and cell state at timestep \(t-1\) , respectively

  • \(\widetilde{c_t}\) represents the candidate cell state

  • \(i_t\) , \(f_t\) and \(o_t\) represent input gate, forget gate, output gate, respectively

  • \(W\) represents weight (e.g., \(W_{ix}\) is the weight of a linear transformation of input \(x_{t}\) when calculating input gate \(i_t\) )

  • \(b\) represents bias (e.g., \(b_{i}\) is the bias of input gate)

  • \(\sigma\) represents nonlinear activation function for gate, default sigmoid

  • \(\odot\) represents the Hadamard product of a matrix, i.e. multiplying the elements of the same position for two matrices with the same dimension to get another matrix with the same dimension

Parameters
  • input (Variable) – LSTM input tensor, multi-dimensional LODTensor of shape \([T, 4*hidden\_size]\) . Data type is float32 or float64.

  • size (int) – must be 4 * hidden_size.

  • h_0 (Variable , optional) – The initial hidden state of the LSTM, multi-dimensional Tensor of shape \([batch\_size, hidden\_size]\) . Data type is float32 or float64. If set to None, it will be a vector of all 0. Default: None.

  • c_0 (Variable , optional) – The initial hidden state of the LSTM, multi-dimensional Tensor of shape \([batch\_size, hidden\_size]\) . Data type is float32 or float64. If set to None, it will be a vector of all 0. h_0 and c_0 can be None but only at the same time. Default: None.

  • param_attr (ParamAttr, optional) –

    Parameter attribute of weight. If it is None, the default weight parameter attribute is used. Please refer to ref:api_fluid_ParamAttr’ . If the user needs to set this parameter, the dimension must be :math:`[hidden_size, 4*hidden_size] . Default: None.

    • Weights = \(\{ W_{cr},W_{ir},W_{fr},W_{or} \}\) , the shape is [hidden_size, 4*hidden_size].

  • bias_attr (ParamAttr, optional) –

    The bias attribute for the learnable bias weights, which contains two parts, input-hidden bias weights and peephole connections weights if setting use_peepholes to True. Please refer to ref:`api_fluid_ParamAttr’ . Default: None.

    System Message: WARNING/2 (/usr/local/lib/python2.7/dist-packages/paddle/fluid/layers/rnn.py:docstring of paddle.fluid.layers.dynamic_lstm, line 51); backlink

    Inline interpreted text or phrase reference start-string without end-string.

    1. use_peepholes = False - Biases = {\(b_c, b_i, b_f, b_o\)}. - The shape is [1, 4*hidden_size].

    2. use_peepholes = True - Biases = { \(b_c, b_i, b_f, b_o, W_{ic}, W_{fc}, W_{oc}\)}. - The shape is [1, 7*hidden_size].

  • use_peepholes (bool, optional) – Whether to use peephole connection or not. Default: True.

  • is_reverse (bool, optional) – Whether to calculate reverse LSTM. Default: False.

  • gate_activation (str, optional) – The activation for input gate, forget gate and output gate. Default: “sigmoid”.

  • cell_activation (str, optional) – The activation for cell output. Default: “tanh”.

  • candidate_activation (str, optional) – The activation for candidate hidden state. Default: “tanh”.

  • dtype (str, optional) – Data type, can be “float32” or “float64”. Default: “float32”.

  • name (str, optional) – A name for this layer. Please refer to Name . Default: None.

Returns

The hidden state and cell state of LSTM

  • hidden: LoDTensor with shape of \([T, hidden\_size]\) , and its lod and dtype is the same as the input.

  • cell: LoDTensor with shape of \([T, hidden\_size]\) , and its lod and dtype is the same as the input.

Return type

tuple ( Variable , Variable )

Examples

import paddle.fluid as fluid
emb_dim = 256
vocab_size = 10000
hidden_dim = 512

data = fluid.data(name='x', shape=[None], dtype='int64', lod_level=1)
emb = fluid.embedding(input=data, size=[vocab_size, emb_dim], is_sparse=True)

forward_proj = fluid.layers.fc(input=emb, size=hidden_dim * 4,
                               bias_attr=False)

forward, cell = fluid.layers.dynamic_lstm(
    input=forward_proj, size=hidden_dim * 4, use_peepholes=False)
forward.shape  # (-1, 512)
cell.shape  # (-1, 512)