hsigmoid_loss

paddle.nn.functional. hsigmoid_loss ( input, label, num_classes, weight, bias=None, path_table=None, path_code=None, is_sparse=False, name=None ) [source]

The hierarchical sigmoid organizes the classes into a complete binary tree to reduce the computational complexity and speed up the model training, especially the training of language model.

Each leaf node of the complete binary tree represents a class(word) and each non-leaf node acts as a binary classifier. For each class(word), there’s a unique path from root to itself, hsigmoid calculate the cost for each non-leaf node on the path, and sum them to get a total cost.

Comparing to softmax, hsigmoid can reduce the computational complexity from \(O(N)\) to \(O(logN)\), where \(N\) represents the number of classes or the size of word dict.

The API supports default tree and custom tree. For the default tree, you can refer to Hierarchical Probabilistic Neural Network Language Model.

For the custom tree, you need to set is_custom to True, and do the following steps (take the language model as an example):

  1. Using a custom word dict to build a binary tree, each leaf node should be an word in the word dict.

  2. Creating a dict map word_id -> path that from the word to the root node, we call it path_table.

  3. Creating a dict map word_id -> code of path that from the word to the root node, we call it path_code. Code means the label of each binary classifier, 1 indicate true, 0 indicate false.

  4. Now, each word should has its path and code along the path, you can pass a batch of path and code related to the same batch of inputs.

Parameters

  • input (Tensor) – A tensor with the shape [N, D], where N is the size of mini-batch, and D is the feature size. Its data type supports float32 or float64.

  • label (Tensor) – A tensor contains the labels of training data. Its shape is [N, 1] and data type is int64.

  • num_classes (int) – The number of classes or the size of word dict, must be greater than 2. If the default tree is used (path_code and path_table is None are None), num_classes should not be None. If the custom tree is used (path_code and path_table is None are not None), num_classes should be the number of non-leaf nodes, which indicates the num of classes using by the binary classifier.

  • weight (Tensor) – A tensor with shape (num_classes - 1, D), with the same data type as input.

  • bias (Tensor, optional) – A tensor with shape (num_classes - 1, 1), with the same data type as input. If bias is None, no bias will be add. Default is None.

  • path_table (Tensor, optional) – A tensor that stores each batch of samples’ path from leaf to root node, its shape is [N, L] and data type is int64, where L is the length of path. For each sample i, path_table[i] is a np.array like structure and each element in this array is the indexes in parent nodes’ weight matrix. If path_table and path_code are None, the default tree will be used. Default is None.

  • path_code (Tensor, optional) – A tensor that stores each batch of samples’ code of path from leaf to root node, its shape is [N, L] and data type is int64, which is the same as path_table. Each code of path is consisted with the code of nodes from leaf to root node. If path_table and path_code are None, the default tree will be used. Default is None.

  • is_sparse (bool, optional) – Whether use sparse updating instead of dense updating. If is_sparse is True, the gradient of weight and input will be sparse. Default is False.

  • name (str, optional) – Name for the operation (optional, default is None). For more information, please refer to Name.

Returns

A tensor with the cost of hierarchical sigmoid, its shape is [N, 1] and data type is the same as input.

Examples

>>> import paddle
>>> import paddle.nn.functional as F

>>> paddle.set_device('cpu')
>>> paddle.seed(2023)

>>> input = paddle.uniform([4, 3])
>>> print(input)
Tensor(shape=[4, 3], dtype=float32, place=Place(cpu), stop_gradient=True,
        [[ 0.73167229,  0.04029441, -0.48078126],
         [ 0.81050646, -0.15199822, -0.18717426],
         [ 0.94041789,  0.48874724,  0.03570259],
         [ 0.46585739,  0.95573163, -0.91368192]])
>>> label = paddle.to_tensor([0, 1, 4, 5])
>>> num_classes = 5
>>> weight = paddle.uniform([num_classes - 1, 3])
>>> print(weight)
Tensor(shape=[4, 3], dtype=float32, place=Place(cpu), stop_gradient=True,
        [[-0.14721161,  0.43916738, -0.58377075],
         [-0.60536981, -0.23151302, -0.70793629],
         [-0.54572451, -0.10784978, -0.56684279],
         [ 0.35370791, -0.07079649,  0.84765708]])
>>> out = F.hsigmoid_loss(input, label, num_classes, weight)
>>> print(out)
Tensor(shape=[4, 1], dtype=float32, place=Place(cpu), stop_gradient=True,
        [[2.23681736],
         [1.97140026],
         [1.66425037],
         [2.54727197]])