Inference Output CPU vs CUDA not the same [1.19.2]

[lschaupp]

Describe the issue

Hey everyone,

I was testing a model for face occlusion and I am getting different results between GPU and CPU.
Happy to help if anyone can give me into the right direction? (e.g. debug) to help fixing this issue.

Cheers

To reproduce

Load the model on the same hardware -> different results.

Urgency

Very important

Platform

Linux

OS Version

20.05

ONNX Runtime Installation

Released Package

ONNX Runtime Version or Commit ID

1.19.2

ONNX Runtime API

Python

Architecture

X64

Execution Provider

CUDA

Execution Provider Library Version

CUDA 12.5

[tianleiwu]

Is it fp16 model? CPU might use fp32 in computation thus will get different accuracy compared to CUDA.

Also CUDA has enabled tf32 by default, you can set environment variable NVIDIA_TF32_OVERRIDE = 0 or set cuda provider option use_tf32 = 0 to disable it.

One way to debug it to build onnxruntime from source and adding --cmake_extra_defines onnxruntime_DEBUG_NODE_INPUTS_OUTPUTS=1 in build command line. Then set environment variables ORT_DEBUG_NODE_IO_DUMP_INPUT_DATA=1 and ORT_DEBUG_NODE_IO_DUMP_OUTPUT_DATA =1. Then you should be able to check and compare the intermediate value between cpu ep and cuda ep. See https://onnxruntime.ai/docs/build/eps.html#cuda for more information