Training on data with very low (<0.05 eV/A) forces - large force errors

[vsumaria]

The question
I am trying to train a potential on a 5 element system. The training is fine overall, but when I try to add very force data to training or use it for predictions, the force errors are really large. On the other hand, the overall energy errors are quite reasonable for the same structures. Has someone else seen this? What might be going wrong? Have attached a parity plot showing this issue.

What did you already try?
I have tried different architectures, task_batch_size_force, force_weights, rmse_threshold_trials. All result in the same problem.

Background
The system consists 5 elements - Alumina substrate with Pt clusters and propene and hydrogen adsorbates.