Scalability and tricks for the continuous transition node

[albertpod]

The Continuous Transition node is kinda underappreciated node that enables learning of unknown transition matrices, whether structured or unstructured. When working with an unstructured matrix, a common approach is to apply a matrix normal prior. However, this method has long-standing issues with scalability.

The key rule here is this one

While it's not immediately obvious, the covariance matrix shown represents a peculiar way of expressing a tensor product of two covariance matrices of A.

To address scalability challenges, there’s a trick (hack) we can employ using meta information. This trick allows us to:

1. Keep track of the full matrix A, while
2. Sacrificing the tensor product that encodes the covariance of A.

The trick is surprisingly simple and does not significantly affect inference performance. 🥁 The solution is to diagonalize the covariance of the outgoing message, using the following approach:

cov_A = mW[1, 1] * Vxmx[1, 1] * diageye(length(mA))  

The reasoning stems from how the covariance matrix is computed for an autoregressive node.

These are just preliminary ideas that require further testing, but the ultimate goal is to scale effectively to large models where the matrix A contains thousands of non-zero entries.

I’m still considering how to handle marginal computation in this setup, will follow up on this at some point. If someone is interested, please let me know.