-
Notifications
You must be signed in to change notification settings - Fork 2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
I have some questions. Can GFlowNet be used for topic modeling? What are the challenges involved? Discussion is welcome. #1
Comments
Good question. Yes, a GFlowNet -- and GFlowNet-EM in particular -- could be used for topic modeling, since a topic model is a particular kind of latent variable model, albeit one with a continuous latent variable. Consider the example of LDA, using the notations
I have tried this for very small topic models on synthetic data -- you can find the code for a proof of concept at https://gist.github.com/malkin1729/88227a1e451596e1ea1fc7d4e0a7ae09 -- but never pursued it further. Curious about what you can do with it, and particularly about whether topic models with more interesting structure in the latent can benefit from the GFlowNet approach. |
As far as I know, some tree-structured neural topic models and nonparametric forest-structured topic models do utilize the stick-breaking process with a mixture of Betas. I am considering whether GFlowNet can be used as an alternative for modeling in these cases. Additionally, for certain popular neural topic models based on VAEs, the assumption is often made that the latent variables follow a Gaussian distribution or a logit Gaussian distribution. I am pondering whether continuous-GFlownet theory can be employed as a replacement in these cases. |
A GFlowNet could be used to sample the posterior over latent topic vectors in nonparametric topic models, indeed. However, I have not seen stick-breaking with mixture of Betas in that literature. Do you have a reference? In my code, I simply used mixture of Betas in parametrizing the sampling of a point in the probability simplex by sequentially “breaking off” probability mass to assign to a chosen topic. |
I apologize, but I believe these papers are related. If they are combined with Gflownet, do you have any thoughts or suggestions? |
There is nothing to apologize for. Thank you for the references. I have worked a little on graph-structured topic models and had seen the first paper before, and I quickly looked at the other two now. They are relevant to structured topic models, of course, but I do not see there the use of learned Beta mixtures as posterior estimators, which is what I had asked about in the comment above. A GFlowNet could be used as an amortized variational posterior in any of these models. |
No description provided.
The text was updated successfully, but these errors were encountered: