[BUG] HMM edges matrix initialization contains NaNs

[asn32]

Describe the bug
Hi, I have been testing out the DenseHMM implementation in Pomegranate v1.0.3 for models with > 50 states, and have occasionally encountered a bug where initializing the model.edges matrix using the model.add_edge results in an edges matrix containing NaNs. The NaNs then propagate through downstream calculations including model.forward_backward and model.predict.

I tracked this down to the following snippet located here:

	if self.edges is None:
		self.edges = torch.empty((n, n), dtype=self.dtype, 
			device=self.device) - inf

where torch.empty sometimes returns an array with NaNs, and NaN - float("inf") = NaN. I think additionally, because in my testing I don't set every entry of the edges matrix manually to a specific probability, subsequent usage of model.edges propagates those NaNs.

To Reproduce
Since the bug (I think) comes from the initialization of a large array using torch.empty, the simplest way I have been able to reproduce it is using the above snippet where n is large (> 50), and then not fill in every edge.

The quickest fix I have found is to just pre-set the matrix with torch.log(torch.zeroes((n,n)).

I'm a big fan of the package, and thank you for all the effort you've put in developing it. Just wanted to put this on your radar.

[jmschrei]

Sorry for the delay and thanks for pointing this out. Would you mind posting a minimal reproducing script? I've been having some trouble reproducing the issue, but I also didn't spend that much time on it.

[MBradbury]

I think I may have hit this issue as well. The problem is that a number of the HMM matrices are initialised by creating the memory with torch.empty and then every entry set to -inf by taking away inf. The problem is that if the uninitialised memory contains any values where x - inf is NaN, then you get the NaN instead of -inf.

The first print will non-deterministically be True and the second will always be True.

import torch

n = 100

a = torch.empty((n, n))
b = a - float("inf")
print(torch.isnan(b).any())

a[0,0] = float("inf")
c = a - float("inf")
print(torch.isnan(c).any())

The fix is to use torch.full instead.

[MBradbury]

Also there is a note that torch.empty will fill the tensor with NaNs if torch.use_deterministic_algorithms() and torch.utils.deterministic.fill_uninitialized_memory are set to True.