scale_factor and offset applied twice

[ELE3E]

I'm running a DAP server of the newest version of pydap==3.2.0 with the latest pydap.handlers.netcdf from the master branch. I've noticed that when you have a netCDF4 file where (some) of the variables have a 'scale_factor' and 'offset' attribute set, they are applied twice.
I did some investigation and I think when actually reading out the values from the netCDF file, it should read the raw value and not the scaled value (which is the default as far as I know).
A temporary fix for me was to remove the 'scale_factor' and 'offset from the attributes to be read from the netCDF file, but I don't think this is the optimal solution.