NCE

class paddle.fluid.dygraph.NCE(name_scope, num_total_classes, sample_weight=None, param_attr=None, bias_attr=None, num_neg_samples=None, sampler='uniform', custom_dist=None, seed=0, is_sparse=False)[source]

This interface is used to construct a callable object of the NCE class. For more details, refer to code examples. It implements the function of the NCE loss function. By default this function uses a uniform distribution for sampling, and it compute and return the noise-contrastive estimation training loss. See Noise-contrastive estimation: A new estimation principle for unnormalized statistical models .

Parameters
  • name_scope (str) – The name of this class.

  • num_total_classes (int) – Total number of classes in all samples

  • param_attr (ParamAttr, optional) – The parameter attribute for learnable weights(Parameter) of nce. If it is set to None or one attribute of ParamAttr, nce will create ParamAttr as param_attr. If the Initializer of the param_attr is not set, the parameter is initialized with Xavier. Default: None.

  • bias_attr (ParamAttr or bool, optional) – The attribute for the bias of nce. If it is set to False, no bias will be added to the output units. If it is set to None or one attribute of ParamAttr, nce will create ParamAttr as bias_attr. If the Initializer of the bias_attr is not set, the bias is initialized zero. Default: None.

  • num_neg_samples (int, optional) – The number of negative classes. The default value is 10.

  • sampler (str, optional) – The sampler used to sample class from negtive classes. It can be ‘uniform’, ‘log_uniform’ or ‘custom_dist’. default: ‘uniform’.

  • custom_dist (float[], optional) – A float[] with size=num_total_classes. It is used when sampler is set to ‘custom_dist’. custom_dist[i] is the probability of i-th class to be sampled. Default: None.

  • seed (int, optional) – The seed used in sampler. Default: 0.

  • is_sparse (bool, optional) – The flag indicating whether to use sparse update. If is_sparse is True, the weight@GRAD and bias@GRAD will be changed to SelectedRows. Default: False.

Attribute:

weight (Parameter): the learnable weights of this layer.

bias (Parameter or None): the learnable bias of this layer.

Returns

None

Examples

import numpy as np
import paddle.fluid as fluid

window_size = 5
dict_size = 20
label_word = int(window_size // 2) + 1
inp_word = np.array([[[1]], [[2]], [[3]], [[4]], [[5]]]).astype('int64')
nid_freq_arr = np.random.dirichlet(np.ones(20) * 1000).astype('float32')

with fluid.dygraph.guard():
    words = []
    for i in range(window_size):
        words.append(fluid.dygraph.base.to_variable(inp_word[i]))

    emb = fluid.Embedding(
        'embedding',
        size=[dict_size, 32],
        param_attr='emb.w',
        is_sparse=False)

    embs3 = []
    for i in range(window_size):
        if i == label_word:
            continue

        emb_rlt = emb(words[i])
        embs3.append(emb_rlt)

    embs3 = fluid.layers.concat(input=embs3, axis=1)
    nce = fluid.NCE('nce',
                 num_total_classes=dict_size,
                 num_neg_samples=2,
                 sampler="custom_dist",
                 custom_dist=nid_freq_arr.tolist(),
                 seed=1,
                 param_attr='nce.w',
                 bias_attr='nce.b')

    nce_loss3 = nce(embs3, words[label_word])
forward(input, label, sample_weight=None)

Defines the computation performed at every call. Should be overridden by all subclasses.

Parameters
  • *inputs (tuple) – unpacked tuple arguments

  • **kwargs (dict) – unpacked dict arguments