The below tables are models enabled by the Intel® Neural Compressor.
| Framework | version | model | Accuracy | Performance/ICX8380/1s4c10ins1bs/throughput(samples/sec) | ||||
|---|---|---|---|---|---|---|---|---|
| INT8 | FP32 | Acc Ratio[(INT8-FP32)/FP32] | INT8 | FP32 | Performance Ratio[INT8/FP32] | |||
| tensorflow | 2.6.0 | densenet121 | 73.55% | 72.89% | 0.91% | 365.23 | 303.21 | 1.20x |
| tensorflow | 2.6.0 | densenet161 | 76.27% | 76.29% | -0.03% | 214.13 | 166.86 | 1.28x |
| tensorflow | 2.6.0 | densenet169 | 74.39% | 74.65% | -0.35% | 291.46 | 245.82 | 1.19x |
| tensorflow | 2.6.0 | efficientnet_b0 | 77.81% | 76.75% | 1.38% | 578.37 | 450.25 | 1.28x |
| tensorflow | 2.6.0 | faster_rcnn_inception_resnet_v2 | 37.65% | 38.33% | -1.77% | 3.49 | 2.16 | 1.61x |
| tensorflow | 2.6.0 | faster_rcnn_inception_resnet_v2_saved | 37.53% | 38.33% | -2.09% | 3.49 | 2.17 | 1.61x |
| tensorflow | 2.6.0 | faster_rcnn_resnet101 | 30.28% | 30.39% | -0.36% | 66.41 | 18.43 | 3.60x |
| tensorflow | 2.6.0 | faster_rcnn_resnet101_saved | 30.37% | 30.39% | -0.07% | 66.24 | 16.45 | 4.03x |
| tensorflow | 2.6.0 | inception_resnet_v2 | 80.44% | 80.40% | 0.05% | 269.03 | 137.25 | 1.96x |
| tensorflow | 2.6.0 | inception_v1 | 70.48% | 69.74% | 1.06% | 2202.64 | 1058.20 | 2.08x |
| tensorflow | 2.6.0 | inception_v2 | 74.36% | 73.97% | 0.53% | 1751.31 | 827.81 | 2.11x |
| tensorflow | 2.6.0 | inception_v3 | 77.28% | 76.75% | 0.69% | 868.06 | 384.17 | 2.26x |
| tensorflow | 2.6.0 | inception_v4 | 80.40% | 80.27% | 0.16% | 569.48 | 197.28 | 2.89x |
| tensorflow | 2.6.0 | mask_rcnn_inception_v2 | 28.54% | 28.72% | -0.63% | 134.03 | 51.12 | 2.62x |
| tensorflow | 2.6.0 | mask_rcnn_inception_v2_ckpt | 28.54% | 28.72% | -0.63% | 134.34 | 51.05 | 2.63x |
| tensorflow | 2.6.0 | mobilenetv1 | 71.79% | 70.96% | 1.17% | 3831.42 | 1189.06 | 3.22x |
| tensorflow | 2.6.0 | mobilenetv2 | 71.79% | 71.76% | 0.04% | 2570.69 | 1237.62 | 2.07x |
| tensorflow | 2.6.0 | resnet101 | 77.50% | 76.45% | 1.37% | 849.62 | 345.54 | 2.46x |
| tensorflow | 2.6.0 | resnet50_fashion | 77.80% | 78.12% | -0.41% | 3344.48 | 1766.78 | 1.90x |
| tensorflow | 2.6.0 | resnet50v1.0 | 74.11% | 74.27% | -0.22% | 1287.00 | 495.29 | 2.60x |
| tensorflow | 2.6.0 | resnet50v1.5 | 76.82% | 76.46% | 0.47% | 1218.03 | 420.34 | 2.90x |
| tensorflow | 2.6.0 | resnetv2_101 | 72.67% | 71.87% | 1.11% | 446.03 | 312.30 | 1.43x |
| tensorflow | 2.6.0 | resnetv2_152 | 73.03% | 72.37% | 0.91% | 308.83 | 214.78 | 1.44x |
| tensorflow | 2.6.0 | resnetv2_50 | 70.33% | 69.64% | 0.99% | 701.26 | 549.75 | 1.28x |
| tensorflow | 2.6.0 | ssd_mobilenet_v1 | 22.97% | 23.13% | -0.69% | 842.46 | 404.04 | 2.08x |
| tensorflow | 2.6.0 | ssd_mobilenet_v1_ckpt | 22.99% | 23.13% | -0.61% | 838.22 | 400.80 | 2.09x |
| tensorflow | 2.6.0 | ssd_resnet34 | 21.69% | 22.09% | -1.81% | 41.23 | 10.75 | 3.83x |
| tensorflow | 2.6.0 | ssd_resnet50_v1 | 37.86% | 38.00% | -0.37% | 65.52 | 24.01 | 2.73x |
| tensorflow | 2.6.0 | ssd_resnet50_v1_ckpt | 37.81% | 38.00% | -0.50% | 66.53 | 21.21 | 3.14x |
| tensorflow | 2.6.0 | transformer_lt | 25.87 | 25.86 | 0.07% | 15.69 | 15.72 | 1.00x |
| tensorflow | 2.6.0 | vgg16 | 72.66% | 70.89% | 2.50% | 642.67 | 166.64 | 3.86x |
| tensorflow | 2.6.0 | vgg19 | 72.72% | 71.01% | 2.41% | 519.48 | 139.24 | 3.73x |
| tensorflow | 2.6.0 | wide_deep_large_ds | 77.62% | 77.67% | -0.07% | 8000.00 | 5263.16 | 1.52x |
| Framework | version | model | Accuracy | Performance/ICX8380/1s4c10ins1bs/throughput(samples/sec) | ||||
|---|---|---|---|---|---|---|---|---|
| INT8 | FP32 | Acc Ratio[(INT8-FP32)/FP32] | INT8 | FP32 | Performance Ratio[INT8/FP32] | |||
| tensorflow | 1.15.0-up3 | bert_large_squad | 92.42 | 92.98 | -0.61% | 23.54 | 11.64 | 2.02x |
| tensorflow | 1.15.0-up3 | bert_base_mrpc | 86.52% | 86.52% | 0.00% | 257.93 | 131.15 | 1.97x |
| tensorflow | 1.15.0-up3 | resnet_v1_50_slim | 76.37% | 75.18% | 1.58% | 1449.28 | 403.71 | 3.59x |
| tensorflow | 1.15.0-up3 | resnet_v1_101_slim | 77.49% | 76.40% | 1.43% | 803.86 | 218.53 | 3.68x |
| tensorflow | 1.15.0-up3 | resnet_v1_152_slim | 77.19% | 76.81% | 0.49% | 568.18 | 144.91 | 3.92x |
| tensorflow | 1.15.0-up3 | inception_v1_slim | 70.49% | 69.77% | 1.03% | 1941.75 | 780.03 | 2.49x |
| tensorflow | 1.15.0-up3 | inception_v2_slim | 74.35% | 73.98% | 0.50% | 1557.63 | 644.33 | 2.42x |
| tensorflow | 1.15.0-up3 | inception_v3_slim | 78.31% | 77.99% | 0.41% | 919.12 | 280.82 | 3.27x |
| tensorflow | 1.15.0-up3 | inception_v4_slim | 80.27% | 80.19% | 0.10% | 504.29 | 140.81 | 3.58x |
| tensorflow | 1.15.0-up3 | vgg16_slim | 72.78% | 70.89% | 2.67% | 582.41 | 142.07 | 4.10x |
| tensorflow | 1.15.0-up3 | vgg19_slim | 72.60% | 71.01% | 2.24% | 492.13 | 118.36 | 4.16x |
| tensorflow | 1.15.0-up3 | resnetv2_50_slim | 70.44% | 69.72% | 1.03% | 773.99 | 469.48 | 1.65x |
| tensorflow | 1.15.0-up3 | resnetv2_101_slim | 72.65% | 71.91% | 1.03% | 460.83 | 249.19 | 1.85x |
| tensorflow | 1.15.0-up3 | resnetv2_152_slim | 73.03% | 72.40% | 0.87% | 326.90 | 168.86 | 1.94x |
| Framework | version | model | Accuracy | Performance/ICX8380/1s4c10ins1bs/throughput(samples/sec) | ||||
|---|---|---|---|---|---|---|---|---|
| INT8 | FP32 | Acc Ratio[(INT8-FP32)/FP32] | INT8 | FP32 | Performance Ratio[INT8/FP32] | |||
| pytorch | 1.9.0+cpu | albert_base_mrpc | 88.77% | 88.50% | 0.31% | 30.84 | 26.48 | 1.16x |
| pytorch | 1.9.0+cpu | barthez_mrpc | 83.51% | 83.81% | -0.35% | 124.82 | 75.98 | 1.64x |
| pytorch | 1.9.0+cpu | bert_base_cola | 59.06% | 58.84% | 0.37% | 198.53 | 105.29 | 1.89x |
| pytorch | 1.9.0+cpu | bert_base_mrpc | 88.12% | 88.73% | -0.69% | 199.32 | 107.34 | 1.86x |
| pytorch | 1.8.0+cpu | bert_base_mrpc_qat | 90% | 89.50% | 0.56% | 183.22 | 102.91 | 1.78x |
| pytorch | 1.9.0+cpu | bert_base_rte | 70.40% | 69.68% | 1.04% | 192.90 | 107.25 | 1.80x |
| pytorch | 1.9.0+cpu | bert_base_sst-2 | 91.74% | 91.86% | -0.13% | 197.86 | 105.31 | 1.88x |
| pytorch | 1.9.0+cpu | bert_base_sts-b | 88.72% | 89.27% | -0.62% | 203.29 | 107.03 | 1.90x |
| pytorch | 1.9.0+cpu | bert_large_cola | 62.07% | 62.57% | -0.80% | 94.97 | 33.77 | 2.81x |
| pytorch | 1.9.0+cpu | bert_large_mrpc | 87.66% | 88.33% | -0.75% | 94.08 | 33.84 | 2.78x |
| pytorch | 1.9.0+cpu | bert_large_qnli | 91.12% | 91.82% | -0.76% | 93.75 | 33.73 | 2.78x |
| pytorch | 1.9.0+cpu | bert_large_rte | 72.20% | 72.56% | -0.50% | 52.80 | 33.62 | 1.57x |
| pytorch | 1.9.0+cpu | bert_large_squad | 92.69 | 93.05 | -0.38% | 20.93 | 11.18 | 1.87x |
| pytorch | 1.9.0+cpu | blendcnn | 68.40% | 68.40% | 0.00% | 5154.64 | 4149.38 | 1.25x |
| pytorch | 1.9.0+cpu | camembert_base_mrpc | 84.31% | 84.22% | 0.11% | 191.17 | 106.92 | 1.79x |
| pytorch | 1.9.0+cpu | ctrl_mrpc | 82.00% | 82.00% | 0.00% | 21.32 | 8.44 | 2.53x |
| pytorch | 1.9.0+cpu | deberta_mrpc | 90.04% | 90.91% | -0.96% | 104.59 | 64.14 | 1.63x |
| pytorch | 1.9.0+cpu | dialogpt_wikitext | 36.18 | 36.18 | 0.00% | 5.78 | 5.72 | 1.01x |
| pytorch | 1.9.0+cpu | distilbert_base_mrpc | 81.17% | 80.99% | 0.21% | 330.25 | 197.78 | 1.67x |
| pytorch | 1.9.0+cpu | flaubert_mrpc | 80.38% | 80.19% | 0.23% | 361.53 | 306.37 | 1.18x |
| pytorch | 1.9.0+cpu | funnel_mrpc | 91.72% | 92.26% | -0.58% | 122.68 | 121.45 | 1.01x |
| pytorch | 1.9.0+cpu | gpt_wikitext | 60.06 | 60.2 | -0.23% | 19.51 | 17.99 | 1.08x |
| pytorch | 1.9.0+cpu | gpt2_mrpc | 83.82% | 83.49% | 0.39% | 113.97 | 114.36 | 1.00x |
| pytorch | 1.9.0+cpu | inception_v3 | 69.48% | 69.54% | -0.09% | 418.59 | 207.77 | 2.01x |
| pytorch | 1.9.0+cpu | layoutlm_mrpc | 81.22% | 78.01% | 4.12% | 178.86 | 100.63 | 1.78x |
| pytorch | 1.9.0+cpu | longformer_mrpc | 90.88% | 91.46% | -0.64% | 20.72 | 15.97 | 1.30x |
| pytorch | 1.9.0+cpu | marianmt_wmt_en_ro | 22.39 | 22.23 | 0.72% | 2.95 | 2.77 | 1.06x |
| pytorch | 1.9.0+cpu | maskrcnn_fx | 37.70% | 37.80% | -0.26% | 84.81 | 51.67 | 1.64x |
| pytorch | 1.9.0+cpu | mbart_wnli | 56.34% | 56.34% | 0.00% | 59.14 | 28.01 | 2.11x |
| pytorch | 1.9.0+cpu | mobilenet_v2 | 70.59% | 71.86% | -1.76% | 658.76 | 494.80 | 1.33x |
| pytorch | 1.9.0+cpu | pegasus_billsum | 50.23 | 51.21 | -1.91% | 0.25 | 0.17 | 1.48x |
| pytorch | 1.9.0+cpu | peleenet | 71.61% | 72.08% | -0.66% | 461.47 | 359.58 | 1.28x |
| pytorch | 1.9.0+cpu | reformer_crime_and_punishment | 6.55 | 6.5 | 0.82% | 214.78 | 209.82 | 1.02x |
| pytorch | 1.9.0+cpu | resnet18 | 69.59% | 69.76% | -0.24% | 692.04 | 363.64 | 1.90x |
| pytorch | 1.9.0+cpu | resnet18_fx | 69.56% | 69.76% | -0.28% | 699.79 | 360.62 | 1.94x |
| pytorch | 1.9.0+cpu | resnet18_qat | 69.75% | 69.76% | -0.02% | 690.97 | 359.46 | 1.92x |
| pytorch | 1.9.0+cpu | resnet18_qat_fx | 69.72% | 69.76% | -0.05% | 695.52 | 352.69 | 1.97x |
| pytorch | 1.9.0+cpu | resnet50 | 76.00% | 76.13% | -0.17% | 453.10 | 186.67 | 2.43x |
| pytorch | 1.9.0+cpu | resnet50_ipex | 75.63% | 76.13% | -0.66% | 495.05 | 298.06 | 1.66x |
| pytorch | 1.9.0+cpu | resnet50_qat | 76.05% | 76.13% | -0.11% | 454.40 | 190.47 | 2.39x |
| pytorch | 1.9.0+cpu | resnext101_32x8d | 79.02% | 79.31% | -0.36% | 196.27 | 70.08 | 2.80x |
| pytorch | 1.9.0+cpu | rnnt | 92.48 | 92.55 | -0.08% | 80.23 | 19.06 | 4.21x |
| pytorch | 1.9.0+cpu | roberta_base_mrpc | 84.70% | 85.51% | -0.94% | 195.39 | 106.28 | 1.84x |
| pytorch | 1.9.0+cpu | se_resnext50_32x4d | 79.06% | 79.08% | -0.02% | 351.74 | 159.64 | 2.20x |
| pytorch | 1.9.0+cpu | squeezebert_mrpc | 87.92% | 87.65% | 0.31% | 169.12 | 160.36 | 1.05x |
| pytorch | 1.9.0+cpu | ssd_resnet34_fx | 19.51 | 19.63 | -0.61% | 28.96 | 7.40 | 3.91x |
| pytorch | 1.9.0+cpu | ssd_resnet34_qat_fx | 17.80% | 17.30% | 2.89% | 274.14 | 98.01 | 2.80x |
| pytorch | 1.9.0+cpu | t5_wmt_en_ro | 24.39 | 24.52 | -0.55% | 4.45 | 4.12 | 1.08x |
| pytorch | 1.9.0+cpu | transfo_xl_mrpc | 82.09% | 81.20% | 1.09% | 9.55 | 7.17 | 1.33x |
| pytorch | 1.9.0+cpu | xlm_mrpc | 80.50% | 79.56% | 1.18% | 53.46 | 18.17 | 2.94x |
| pytorch | 1.9.0+cpu | xlm_roberta_mrpc | 88.24% | 88.24% | 0.00% | 98.21 | 98.28 | 1.00x |
| pytorch | 1.9.0+cpu | xlm-roberta-base_mrpc | 88.15% | 88.62% | -0.53% | 98.95 | 99.85 | 0.99x |
| pytorch | 1.9.0+cpu | xlnet_base_mrpc | 89.43% | 89.47% | -0.04% | 87.80 | 68.69 | 1.28x |
| pytorch | 1.9.0+cpu | yolo_v3 | 24.50% | 24.54% | -0.17% | 98.11 | 37.50 | 2.62x |
| Framework | version | model | Accuracy | Performance/ICX8380/1s4c10ins1bs/throughput(samples/sec) | ||||
|---|---|---|---|---|---|---|---|---|
| INT8 | FP32 | Acc Ratio[(INT8-FP32)/FP32] | INT8 | FP32 | Performance Ratio[INT8/FP32] | |||
| mxnet | 1.7.0 | inceptionv3 | 77.73% | 77.64% | 0.11% | 974.90 | 274.73 | 3.55x |
| mxnet | 1.7.0 | mobilenet1.0 | 71.69% | 72.22% | -0.74% | 7235.37 | 2507.08 | 2.89x |
| mxnet | 1.7.0 | mobilenetv2_1.0 | 70.78% | 70.87% | -0.12% | 5961.96 | 2156.01 | 2.77x |
| mxnet | 1.7.0 | resnet152_v1 | 78.21% | 78.54% | -0.42% | 563.41 | 149.23 | 3.78x |
| mxnet | 1.7.0 | resnet18_v1 | 70.02% | 70.14% | -0.17% | 3537.57 | 807.82 | 4.38x |
| mxnet | 1.7.0 | resnet50v1 | 76.08% | 76.33% | -0.32% | 1587.63 | 415.49 | 3.82x |
| mxnet | 1.7.0 | squeezenet1.0 | 56.74% | 56.96% | -0.38% | 4911.11 | 1487.43 | 3.30x |
| mxnet | 1.7.0 | ssd-mobilenet1.0 | 74.94% | 75.54% | -0.79% | 738.51 | 174.83 | 4.22x |
| mxnet | 1.7.0 | ssd-resnet50_v1 | 80.21% | 80.23% | -0.03% | 268.91 | 54.82 | 4.91x |
| Framework | version | model | Accuracy | Performance/ICX8380/1s4c10ins1bs/throughput(samples/sec) | ||||
|---|---|---|---|---|---|---|---|---|
| INT8 | FP32 | Acc Ratio[(INT8-FP32)/FP32] | INT8 | FP32 | Performance Ratio[INT8/FP32] | |||
| onnxrt | 1.8.0 | alexnet | 54.68% | 54.80% | -0.22% | 1195.53 | 626.44 | 1.91x |
| onnxrt | 1.8.0 | bert_base_mrpc_dynamic | 84.56% | 86.03% | -1.71% | 341.47 | 144.42 | 2.36x |
| onnxrt | 1.8.0 | bert_base_mrpc_static | 85.29% | 86.03% | -0.86% | 683.80 | 294.99 | 2.32x |
| onnxrt | 1.8.0 | bert_squad_model_zoo | 80.43 | 80.67 | -0.29% | 106.91 | 59.97 | 1.78x |
| onnxrt | 1.8.0 | caffenet | 56.22% | 56.27% | -0.09% | 1739.77 | 564.82 | 3.08x |
| onnxrt | 1.8.0 | distilbert_base_mrpc | 84.56% | 84.56% | 0.00% | 1626.07 | 554.50 | 2.93x |
| onnxrt | 1.8.0 | googlenet-12 | 67.73% | 67.78% | -0.07% | 928.78 | 717.07 | 1.30x |
| onnxrt | 1.8.0 | gpt2_lm_head_wikitext_model_zoo | 32.07 | 28.99 | 10.61% | 1.46 | 1.30 | 1.12x |
| onnxrt | 1.8.0 | mobilebert_mrpc | 84.31% | 86.27% | -2.27% | 766.17 | 649.96 | 1.18x |
| onnxrt | 1.8.0 | mobilebert_squad_mlperf | 89.84 | 90.02 | -0.20% | 91.06 | 81.05 | 1.12x |
| onnxrt | 1.8.0 | mobilenet_v2 | 65.19% | 66.92% | -2.59% | 2678.31 | 2807.88 | 0.95x |
| onnxrt | 1.8.0 | mobilenet_v3_mlperf | 75.51% | 75.75% | -0.32% | 2960.51 | 1881.47 | 1.57x |
| onnxrt | 1.8.0 | resnet_v1_5_mlperf | 76.07% | 76.47% | -0.52% | 884.23 | 497.15 | 1.78x |
| onnxrt | 1.8.0 | resnet50_v1_5 | 72% | 72% | 0% | 855.49 | 493.39 | 1.73x |
| onnxrt | 1.8.0 | resnet50-v1-12 | 74.83% | 74.97% | -0.19% | 1008.72 | 520.80 | 1.94x |
| onnxrt | 1.8.0 | roberta_base_mrpc | 88.24% | 89.46% | -1.36% | 724.68 | 284.01 | 2.55x |
| onnxrt | 1.8.0 | shufflenet-v2-12 | 66.15% | 66.35% | -0.30% | 4502.48 | 2721.01 | 1.65x |
| onnxrt | 1.8.0 | squeezenet | 56.48% | 56.85% | -0.65% | 5008.01 | 3629.11 | 1.38x |
| onnxrt | 1.8.0 | ssd_mobilenet_v1 | 22.47% | 23.10% | -2.73% | 730.17 | 627.50 | 1.16x |
| onnxrt | 1.8.0 | ssd_mobilenet_v2 | 23.90% | 24.68% | -3.16% | 558.03 | 446.69 | 1.25x |
| onnxrt | 1.8.0 | vgg16 | 66.55% | 66.68% | -0.19% | 145.10 | 122.70 | 1.18x |
| onnxrt | 1.8.0 | vgg16_model_zoo | 72.32% | 72.38% | -0.08% | 253.32 | 121.09 | 2.09x |
| onnxrt | 1.8.0 | zfnet | 55.84% | 55.97% | -0.23% | 536.71 | 336.96 | 1.59x |