Embedding

class paddle.nn. Embedding ( num_embeddings, embedding_dim, padding_idx=None, sparse=False, weight_attr=None, name=None ) [source]

Embedding Layer

This interface is used to construct a callable object of the Embedding class. For specific usage, refer to code examples. It implements the function of the Embedding Layer. This layer is used to lookup embeddings vector of ids provided by x . It automatically constructs a 2D embedding matrix based on the input num_embeddings and embedding_dim.

The shape of output Tensor is generated by appending an emb_size dimension to the last dimension of the input Tensor shape.

Note: The id in x must satisfy \(0 =< id < num_embeddings\) , otherwise the program will throw an exception and exit.

Case 1:

x is a Tensor. padding_idx = -1
    x.data = [[1, 3], [2, 4], [4, 127]
    x.shape = [3, 2]
Given size = [128, 16]
output is a Tensor:
    out.shape = [3, 2, 16]
    out.data = [[[0.129435295, 0.244512452, ..., 0.436322452],
                [0.345421456, 0.524563927, ..., 0.144534654]],

                [[0.345249859, 0.124939536, ..., 0.194353745],
                [0.945345345, 0.435394634, ..., 0.435345365]],

                [[0.945345345, 0.435394634, ..., 0.435345365],
                [0.0,         0.0,         ..., 0.0        ]]]  # padding data
The input padding_idx is less than 0, it is automatically converted to padding_idx = -1 + 128 = 127
It will pad all-zero data when ids is 127.
Parameters
  • num_embeddings (int) – Just one element which indicate the size of the dictionary of embeddings.

  • embedding_dim (int) – Just one element which indicate the size of each embedding vector respectively.

  • padding_idx (int|long|None) – padding_idx needs to be in the interval [-num_embeddings, num_embeddings). If \(padding\_idx < 0\), the \(padding\_idx\) will automatically be converted to \(vocab\_size + padding\_idx\) . It will output all-zero padding data whenever lookup encounters \(padding\_idx\) in id. And the padding data will not be updated while training. If set None, it makes no effect to output. Default: None.

  • sparse (bool) – The flag indicating whether to use sparse update. This parameter only affects the performance of the backwards gradient update. It is recommended to set True because sparse update is faster. But some optimizer does not support sparse update, such as Adadelta , Adamax , Lamb. In these case, sparse must be False. Default: False.

  • weight_attr (ParamAttr) – To specify the weight parameter property. Default: None, which means the default weight parameter property is used. See usage for details in api_ParamAttr . In addition, user-defined or pre-trained word vectors can be loaded with the param_attr parameter. The local word vector needs to be transformed into numpy format, and the shape of local word vector should be consistent with num_embeddings . Then api_initializer_NumpyArrayInitializer is used to load custom or pre-trained word vectors. See code example for details.

  • name (str|None) – For detailed information, please refer to Name. Usually name is no need to set and None by default.

Attribute:

weight (Parameter): the learnable weights of this layer.

Returns

None

Examples

import paddle
import numpy as np

x_data = np.arange(3, 6).reshape((3, 1)).astype(np.int64)
y_data = np.arange(6, 12).reshape((3, 2)).astype(np.float32)

x = paddle.to_tensor(x_data, stop_gradient=False)
y = paddle.to_tensor(y_data, stop_gradient=False)

embedding = paddle.nn.Embedding(10, 3, sparse=True)

w0=np.full(shape=(10, 3), fill_value=2).astype(np.float32)
embedding.weight.set_value(w0)

adam = paddle.optimizer.Adam(parameters=[embedding.weight], learning_rate=0.01)
adam.clear_grad()

# weight.shape = [10, 3]

# x.data = [[3],[4],[5]]
# x.shape = [3, 1]

# out.data = [[2,2,2], [2,2,2], [2,2,2]]
# out.shape = [3, 1, 3]
out=embedding(x)
out.backward()
adam.step()
forward ( x )

Defines the computation performed at every call. Should be overridden by all subclasses.

Parameters
  • *inputs (tuple) – unpacked tuple arguments

  • **kwargs (dict) – unpacked dict arguments

extra_repr ( )

Extra representation of this layer, you can have custom implementation of your own layer.