feat: alt-text auto generation

[Chanbinski]

The code doesn't work but created a PR for code review.

[gmerritt]

@Chanbinski, I took a quick check of the code so far, with an interest in identifying how it was doing the alt text generation. I find ollama serve & for the local shared binary, but I do not yet find in the code where any specific model is referenced; presumably would need something from like the llava series or perhaps a phi vision model?

Have you found a way to put the model in a shared location for all users in our jupyterhub / container context?

[Chanbinski]

Thanks @gmerritt for the review. This code is intended to run on the JupyterHub and assumes that

1. The ollama binary (llava) is installed in /shared/jupyterlab-a11y-checker/ for all users.

2. Berkeley Datahub has deployed config changes that allows users to access the ollama endpoint by through (/user//ollama). In the index.ts file, this is reflected as "https://a11y-staging.datahub.berkeley.edu/user/chanbin.park/ollama/api/generate" which is my endpoint.

I was testing on a hub where the ollama binary was installed in the right place. The issue is that the code doesn't run ollama that is installed when the extension is launched.

[gmerritt]

@Chanbinski, I believe your are conflating the separate, distinct notions of the framework vs. the model?

Ollama is an application that implements a framework that can load & host models and present a chat completion API. The binary you reference is just the framework. You have to load a model.

Even if the binary for the ollama application is in a shared location, when you ask ollama (via the api) to do a chat completion with a certain model, ollama will load that model in user filespace. There is no way to tell ollama to use a shared filespace for its models.

Please send a reference if you have found otherwise -- but I spent hours on this, it was very frustrating.

Edit: see addendum in comment to follow!

[gmerritt]

Oh, and to refine -- I believe that the real deal-breaker that ollama insists to write to the model location directory. I was mis-remembering the exact issue!

[Chanbinski]

@gmerritt Got it, thanks for letting me know. I will let you know if I come up with any solutions to that.

But the problem prior to that is that the extension doesn't call the function to start ollama in the first place. I've added code (that can be seen in the latest commit) so that the extension serves as a jupyter server extension, but it seems like it's not calling _load_jupyter_server_extension (from init.py) when the extension is first launched. I've followed the exact process from the juputer server docs and referred to a similar PR, but it seems like there is a problem.