Bigger model with slower inference time

[yro7]

Hello,
I'm a beginner in machine learning and I mess with some parameters to get used to the library.

I have two classic MNIST recognition models:

The first is the one from the beginner tutorial, builded as model.setBlock(new Mlp(28 * 28, 10, new int[] {128, 64}));
The second one has another 32 neurons layer : model.setBlock(new Mlp(28 * 28, 10, new int[] {128, 64, 32}));

I have those inferences duration:

MODEL 1 : 0.040713043 seconds 
MODEL 2: 0.00260968 seconds

For what I understand of ML, model2 should have an higher inference time since computing the third layer takes time.
Both models were trained with the same Trainer, on the same number of epochs, same data etc...

What can cause such a difference ? I'm suspecting that it might be the JVM that optimizes the code after running the first part, but the difference seems way too high.