In terms of evaluating the performance of a specific model, we should have general metrics to measure the performance of different models. Different frameworks always have their own Metric module but with different features and APIs. Neural Compressor Metrics supports code-free configuration through a yaml file, with built-in metrics, so that Neural Compressor can achieve performance and accuracy without code changes from the user. In special cases, users can also register their own metric classes through below method.
Users can specify an Neural Compressor built-in metric such as shown below:
evaluation:
accuracy:
metric:
topk: 1Users can also register their own metric as follows:
class Metric(object):
def __init__(self):
# init code here
def update(self, preds, labels):
# add preds and labels to storage
def reset(self):
# clear preds and labels storage
def result(self):
# calculate accuracy
return accuracyThe result() function returns a higher-is-better scalar to reflect model accuracy on an evaluation dataset.
After defining the metric class, users need to register it with a user-defined metric name and the metric class:
from neural_compressor.quantization import Quantization, common
quantizer = Quantization(yaml_file)
quantizer.model = graph
quantizer.metric = common.Metric(NewMetric, 'metric_name')
quantizer.calib_dataloader = dataloader
q_model = quantizer.fit()Neural Compressor supports some built-in metrics that are popularly used in industry.
Refer to this HelloWorld example on how to config a built-in metric.
| Metric | Parameters | Inputs | Comments | Usage(In yaml file) |
|---|---|---|---|---|
| topk(k) | k (int, default=1): Number of top elements to look at for computing accuracy | preds, labels | Computes top k predictions accuracy. | metric: topk: k: 1 |
| Accuracy() | None | preds, labels | Computes accuracy classification score. | metric: Accuracy: {} |
| Loss() | None | preds, labels | A dummy metric for directly printing loss, it calculates the average of predictions. Please refer to MXNet docs for details. |
metric: Loss: {} |
| MAE(compare_label) | compare_label (bool, default=True): Whether to compare label. False if there are no labels and will use FP32 preds as labels | preds, labels | Computes Mean Absolute Error (MAE) loss. | metric: MAE: {} |
| RMSE(compare_label) | compare_label (bool, default=True): Whether to compare label. False if there are no labels and will use FP32 preds as labels | preds, labels | Computes Root Mean Square Error (RMSE) loss. | metric: RMSE: {} |
| MSE(compare_label) | compare_label (bool, default=True): Whether to compare label. False if there are no labels and will use FP32 preds as labels | preds, labels | Computes Mean Squared Error (MSE) loss. | metric: MSE: {} |
| F1() | None | preds, labels | Computes the F1 score of a binary classification problem. | metric: F1: {} |
| mAP(anno_path, iou_thrs, map_points) | anno_path (str): Annotation path. The annotation file should be a yaml file, please refer to label_map for its format. iou_thrs (float or str, default=0.5): Minimal value for intersection over union that allows to make decision that prediction bounding box is true positive. You can specify one float value between 0 to 1 or string "05:0.05:0.95" for standard COCO thresholds. map_points (int, default=0): The way to calculate mAP. 101 for 101-point interpolated AP, 11 for 11-point interpolated AP, 0 for area under PR curve. |
preds, labels | preds is a tuple which supports 2 length: 3 and 4. If its length is 3, it should contain boxes, scores, classes in turn. If its length is 4, it should contain target_boxes_num, boxes, scores, classes in turn labels is a tuple which contains bbox, str_label, int_label, image_id inturn the length of one of str_label and int_label can be 0 |
metric: mAP: anno_path: /path/to/annotation iou_thrs: 0.5 map_points: 0 If anno_path is not set, metric will use official coco label id |
| COCOmAP(anno_path, iou_thrs, map_points) | anno_path (str): Annotation path. The annotation file should be a yaml file, please refer to label_map for its format. iou_thrs (float or str): Intersection over union threshold. Set to "0.5:0.05:0.95" for standard COCO thresholds. map_points (int): The way to calculate mAP. Set to 101 for 101-point interpolated AP. |
preds, labels | preds is a tuple which supports 2 length: 3 and 4. If its length is 3, it should contain boxes, scores, classes in turn. If its length is 4, it should contain target_boxes_num, boxes, scores, classes in turn labels is a tuple which contains bbox, str_label, int_label, image_id inturn the length of one of str_label and int_label can be 0 |
metric: COCOmAP: anno_path: /path/to/annotation If anno_path is not set, metric will use official coco label id |
| VOCmAP(anno_path, iou_thrs, map_points) | anno_path(str): Annotation path. The annotation file should be a yaml file, please refer to label_map for its format. iou_thrs(float or str): Intersection over union threshold. Set to 0.5. map_points(int): The way to calculate mAP. The way to calculate mAP. Set to 0 for area under PR curve. |
preds, labels | preds is a tuple which supports 2 length: 3 and 4. If its length is 3, it should contain boxes, scores, classes in turn. If its length is 4, it should contain target_boxes_num, boxes, scores, classes in turn labels is a tuple which contains bbox, str_label, int_label, image_id inturn the length of one of str_label and int_label can be 0 |
metric: VOCmAP: anno_path: /path/to/annotation If anno_path is not set, metric will use official coco label id |
| COCOmAPv2(anno_path, iou_thrs, map_points, output_index_mapping) | anno_path (str): Annotation path. The annotation file should be a yaml file, please refer to label_map for its format. iou_thrs (float or str): Intersection over union threshold. Set to "0.5:0.05:0.95" for standard COCO thresholds. map_points (int): The way to calculate mAP. Set to 101 for 101-point interpolated AP. output_index_mapping(dict, default={'num_detections':-1, 'boxes':0, 'scores':1, 'classes':2}): Specifies the index of outputs in model raw prediction, -1 means this output does not exist. |
preds, labels | preds is a tuple which supports 2 length: 3 and 4. If its length is 3, it should contain boxes, scores, classes in turn. If its length is 4, it should contain target_boxes_num, boxes, scores, classes in turn labels is a tuple which contains bbox, str_label, int_label, image_id inturn the length of one of str_label and int_label can be 0 |
metric: COCOmAP: anno_path: /path/to/annotation output_index_mapping: num_detections: 0 boxes: 1 scores: 2 classes: 3 If anno_path is not set, metric will use official coco label id |
| BLEU() | None | preds, labels | BLEU score computation between labels and predictions. An approximate BLEU scoring method since we do not glue word pieces or decode the ids and tokenize the output. By default, we use ngram order of 4 and use brevity penalty. Also, this does not have beam search | metric: BLEU: {} |
| SquadF1() | None | preds, labels | Evaluate v1.1 of the SQuAD dataset | metric: SquadF1: {} |
| Metric | Parameters | Inputs | Comments | Usage(In yaml file) |
|---|---|---|---|---|
| topk(k) | k (int, default=1): Number of top elements to look at for computing accuracy | preds, labels | Calculates the top-k categorical accuracy. | metric: topk: k: 1 |
| Accuracy() | None | preds, labels | Calculates the accuracy for binary, multiclass and multilabel data. Please refer Pytorch docs for details. |
metric: Accuracy: {} |
| Loss() | None | preds, labels | A dummy metric for directly printing loss, it calculates the average of predictions. Please refer MXNet docs for details. |
metric: Loss: {} |
| MAE(compare_label) | compare_label(bool, default=True): Whether to compare label. False if there are no labels and will use FP32 preds as labels. | preds, labels | Computes Mean Absolute Error (MAE) loss. | metric: MAE: compare_label： True |
| RMSE(compare_label) | compare_label(bool, default=True): Whether to compare label. False if there are no labels and will use FP32 preds as labels. | preds, labels | Computes Root Mean Squared Error (RMSE) loss. | metric: RMSE: compare_label: True |
| MSE(compare_label) | compare_label(bool, default=True): Whether to compare label. False if there are no labels and will use FP32 preds as labels. | preds, labels | Computes Mean Squared Error (MSE) loss. | metric: MSE: compare_label: True |
| F1() | None | preds, labels | Computes the F1 score of a binary classification problem. | metric: F1: {} |
| Metric | Parameters | Inputs | Comments | Usage(In yaml file) |
|---|---|---|---|---|
| topk(k) | k (int, default=1): Number of top elements to look at for computing accuracy | preds, labels | Computes top k predictions accuracy. | metric: topk: k: 1 |
| Accuracy() | None | preds, labels | Computes accuracy classification score. Please refer to MXNet docs for details. |
metric: Accuracy: {} |
| Loss() | None | preds, labels | A dummy metric for directly printing loss, it calculates the average of predictions. Please refer to MXNet docs for details. |
metric: Loss: {} |
| MAE() | None | preds, labels | Computes Mean Absolute Error (MAE) loss. Please refer to MXNet docs for details. |
metric: MAE: {} |
| RMSE(compare_label) | compare_label(bool, default=True): Whether to compare label. False if there are no labels and will use FP32 preds as labels. | preds, labels | Computes Root Mean Squared Error (RMSE) loss. | metric: RMSE: compare_label: True |
| MSE() | None | preds, labels | Computes Mean Squared Error (MSE) loss. Please refer to MXNet docs for details. |
metric: MSE: {} |
| F1() | None | preds, labels | Computes the F1 score of a binary classification problem. Please refer to MXNet docs for details. |
metric: F1: {} |
| Metric | Parameters | Inputs | Comments | Usage(In yaml file) |
|---|---|---|---|---|
| topk(k) | k (int, default=1): Number of top elements to look at for computing accuracy | preds, labels | Computes top k predictions accuracy. | metric: topk: k: 1 |
| Accuracy() | None | preds, labels | Computes accuracy classification score. | metric: Accuracy: {} |
| Loss() | None | preds, labels | A dummy metric for directly printing loss, it calculates the average of predictions. Please refer to MXNet docs for details. |
metric: Loss: {} |
| MAE(compare_label) | compare_label(bool, default=True): Whether to compare label. False if there are no labels and will use FP32 preds as labels. | preds, labels | Computes Mean Absolute Error (MAE) loss. | metric: MAE: compare_label： True |
| RMSE(compare_label) | compare_label(bool, default=True): Whether to compare label. False if there are no labels and will use FP32 preds as labels. | preds, labels | Computes Root Mean Squared Error (RMSE) loss. | metric: RMSE: compare_label: True |
| MSE(compare_label) | compare_label(bool, default=True): Whether to compare label. False if there are no labels and will use FP32 preds as labels. | preds, labels | Computes Mean Squared Error (MSE) loss. | metric: MSE: compare_label: True |
| F1() | None | preds, labels | Computes the F1 score of a binary classification problem. | metric: F1: {} |
| mAP(anno_path, iou_thrs, map_points) | anno_path(str): Annotation path. The annotation file should be a yaml file, please refer to label_map for its format. The annotation file should be a yaml file, please refer to label_map for its format. iou_thrs(float or str, default=0.5): Minimal value for intersection over union that allows to make decision that prediction bounding box is true positive. You can specify one float value between 0 to 1 or string "05:0.05:0.95" for standard COCO thresholds. map_points(int, default=0): The way to calculate mAP. 101 for 101-point interpolated AP, 11 for 11-point interpolated AP, 0 for area under PR curve. |
preds, labels | preds is a tuple which supports 2 length: 3 and 4. If its length is 3, it should contain boxes, scores, classes in turn. If its length is 4, it should contain target_boxes_num, boxes, scores, classes in turn labels is a tuple which contains bbox, str_label, int_label, image_id inturn the length of one of str_label and int_label can be 0 |
metric: mAP: anno_path: /path/to/annotation iou_thrs: 0.5 map_points: 0 If anno_path is not set, metric will use official coco label id |
| COCOmAP(anno_path, iou_thrs, map_points) | anno_path(str): Annotation path. The annotation file should be a yaml file, please refer to label_map for its format. The annotation file should be a yaml file, please refer to label_map for its format. iou_thrs(float or str): Intersection over union threshold. Set to "0.5:0.05:0.95" for standard COCO thresholds. map_points(int): The way to calculate mAP. Set to 101 for 101-point interpolated AP. |
preds, labels | preds is a tuple which supports 2 length: 3 and 4. If its length is 3, it should contain boxes, scores, classes in turn. If its length is 4, it should contain target_boxes_num, boxes, scores, classes in turn labels is a tuple which contains bbox, str_label, int_label, image_id inturn the length of one of str_label and int_label can be 0 |
metric: COCOmAP: anno_path: /path/to/annotation If anno_path is not set, metric will use official coco label id |
| VOCmAP(anno_path, iou_thrs, map_points) | anno_path(str): Annotation path. The annotation file should be a yaml file, please refer to label_map for its format. The annotation file should be a yaml file, please refer to label_map for its format . iou_thrs(float or str): Intersection over union threshold. Set to 0.5. map_points(int): The way to calculate mAP. The way to calculate mAP. Set to 0 for area under PR curve. |
preds, labels | preds is a tuple which supports 2 length: 3 and 4. If its length is 3, it should contain boxes, scores, classes in turn. If its length is 4, it should contain target_boxes_num, boxes, scores, classes in turn labels is a tuple which contains bbox, str_label, int_label, image_id inturn the length of one of str_label and int_label can be 0 |
metric: VOCmAP: anno_path: /path/to/annotation If anno_path is not set, metric will use official coco label id |
| COCOmAPv2(anno_path, iou_thrs, map_points, output_index_mapping) | anno_path(str): Annotation path. The annotation file should be a yaml file, please refer to label_map for its format. The annotation file should be a yaml file, please refer to label_map for its format. iou_thrs(float or str): Intersection over union threshold. Set to "0.5:0.05:0.95" for standard COCO thresholds. map_points(int): The way to calculate mAP. Set to 101 for 101-point interpolated AP. output_index_mapping(dict, default={'num_detections':-1, 'boxes':0, 'scores':1, 'classes':2}): Specifies the index of outputs in model raw prediction, -1 means this output does not exist. |
preds, labels | preds is a tuple which supports 2 length: 3 and 4. If its length is 3, it should contain boxes, scores, classes in turn. If its length is 4, it should contain target_boxes_num, boxes, scores, classes in turn labels is a tuple which contains bbox, str_label, int_label, image_id inturn the length of one of str_label and int_label can be 0 |
metric: COCOmAP: anno_path: /path/to/annotation output_index_mapping: num_detections: 0 boxes: 1 scores: 2 classes: 3 If anno_path is not set, metric will use official coco label id |
| GLUE(task) | task (str, default=mrpc): The name of the task. Choices include mrpc, qqp, qnli, rte, sts-b, cola, mnli, wnli. | preds, labels | Computes GLUE score for bert model. | metric: GLUE: task: mrpc |