Small speedup when calculating matrices without using derivatives #2967

Job	Run time
codecheck / codecheck	44m 22s
docker / docker (fedora39)	20m 49s
linux / linux (-doc-mpi-)	39m 46s
docker / docker (rocky8)	23m 19s
mac / macports (macos-12)	13m 31s
mac / macsimple (macos-12)	8m 44s
linux / linux (-mpi-)	48m 37s
mac / macports (macos-14)	10m 32s
mac / macsimple (macos-14)	4m 26s
linux / linux (-coverage-mpi-)	1h 2m 3s
mac / macports (+allmodules, macos-12)	20m 9s
linux / linux (-debug-)	31m 0s
conda / conda (ubuntu-20.04)	10m 41s
linux / linux (-debug-mpi-)	39m 52s
conda / conda (macos-12)	17m 7s
linux / linux (-intel-)	44m 33s
conda / conda (macos-14)	7m 15s
linux / linux (-pycv-mpi-)	18m 14s
	7h 45m 0s

Provide feedback