Metric¶

class paddle.metric. Metric [source]

Base class for metric, encapsulates metric logic and APIs Usage:

m = SomeMetric()
for prediction, label in ...:
    m.update(prediction, label)
m.accumulate()

Advanced usage for compute:

Metric calculation can be accelerated by calculating metric states from model outputs and labels by build-in operators not by Python/NumPy in compute, metric states will be fetched as NumPy array and call update with states in NumPy format. Metric calculated as follows (operations in Model and Metric are indicated with curly brackets, while data nodes not):

   inputs & labels              || ------------------
         |                      ||
      {model}                   ||
         |                      ||
  outputs & labels              ||
         |                      ||    tensor data
  {Metric.compute}              ||
         |                      ||
metric states(tensor)           ||
         |                      ||
  {fetch as numpy}              || ------------------
         |                      ||
metric states(numpy)            ||    numpy data
         |                      ||
  {Metric.update}               \/ ------------------

Examples

For Accuracy metric, which takes pred and label as inputs, we can calculate the correct prediction matrix between pred and label in compute. For examples, prediction results contains 10 classes, while pred shape is [N, 10], label shape is [N, 1], N is mini-batch size, and we only need to calculate accurary of top-1 and top-5, we could calculate the correct prediction matrix of the top-5 scores of the prediction of each sample like follows, while the correct prediction matrix shape is [N, 5].

def compute(pred, label):
    # sort prediction and slice the top-5 scores
    pred = paddle.argsort(pred, descending=True)[:, :5]
    # calculate whether the predictions are correct
    correct = pred == label
    return paddle.cast(correct, dtype='float32')

With the compute, we split some calculations to OPs (which may run on GPU devices, will be faster), and only fetch 1 tensor with shape as [N, 5] instead of 2 tensors with shapes as [N, 10] and [N, 1]. update can be define as follows:

def update(self, correct):
    accs = []
    for i, k in enumerate(self.topk):
        num_corrects = correct[:, :k].sum()
        num_samples = len(correct)
        accs.append(float(num_corrects) / num_samples)
        self.total[i] += num_corrects
        self.count[i] += num_samples
    return accs

abstract reset ( )

reset¶

Reset states and result

abstract update ( *args )

update¶

Update states for metric

Inputs of update is the outputs of Metric.compute, if compute is not defined, the inputs of update will be flatten arguments of output of mode and label from data: update(output1, output2, ..., label1, label2,...)

see Metric.compute

abstract accumulate ( )

accumulate¶

Accumulates statistics, computes and returns the metric value

abstract name ( )

name¶

Returns metric name

compute ( *args )

compute¶

This API is advanced usage to accelerate metric calculating, calulations from outputs of model to the states which should be updated by Metric can be defined here, where Paddle OPs is also supported. Outputs of this API will be the inputs of “Metric.update”.

If compute is defined, it will be called with outputs of model and labels from data as arguments, all outputs and labels will be concatenated and flatten and each filed as a separate argument as follows: compute(output1, output2, ..., label1, label2,...)

If compute is not defined, default behaviour is to pass input to output, so output format will be: return output1, output2, ..., label1, label2,...

see Metric.update