Add NASA Earthdata GIS Rest API into QGIS image

[freitagb]

Context

Both VEDA and GHG jupyterhubs have deployed QGIS images for their respective jupyterhubs. We would like tighter integration of those hubs with the NASA Earthdata GIS data catalog. The Earthdata GIS layers are availble via a Rest API and a connection can currently be established manually. It would be great if the Rest API connection was preloaded in the QGIS image so the Earthdata catalog would be available by default.

Proposal

No response

Updates and actions

No response

[freitagb]

here is the Rest API url: https://gis.earthdata.nasa.gov/image/rest/services

[yuvipanda]

I'd love for help from devseed folks here, as this basically is just qgis work. I know they have folks in there with qgis expertise, and it would be extremely helpful here. The alternative is that I basically dive into learning more about qgis itself, and I'm not sure how that would work from a resourcing / timing perspective.

[freitagb]

@j08lue ☝️ can you take a look at the request above?

[batpad]

@yuvipanda are you able to provide a bit more context and specificity on the work involved here and links to existing repositories, etc? I'll take a look and speak to folks.

[j08lue]

I think there are two questions:

1. What kind of integration of the EGIS image services into QGIS are we aiming for? Is there an existing QGIS plugin / mechanism for users to discover these sources, which we just need to install and configure?
2. How do we make the same setup available upon launch of the JupyterHub QGIS images?

Do you have the details about the first, @freitagb? I will follow up on Slack about this.

[yuvipanda]

Some information that may be useful.

1. We're currenly running this fork of the stac plugin (https://github.com/slesaad/qgis-stac-plugin/tree/support_gdal_assets) to allow qgis to pick up AWS credentials automatically. It is basically just bringing in a few unmerged PRs from upstream (https://github.com/stac-utils/qgis-stac-plugin) that was needed because upstream looks dead :( I don't know what the long term plan for this would be - currently, just by dint of owning the repo, @slesaad is on the hook for it and idk if that's intentional :D
2. The image itself if coming from https://github.com/jupyterhub/jupyter-remote-desktop-proxy/tree/base-image/images/qgis. I'll probably move it to its own repo, as we're installing custom forks of qgis plugins and stuff so it's probably not the best fit for an upstream image.

So I think whatever integration is needed, would look like:

1. Figuring out what combination of qgis plugins + config is needed to make that happen locally, on someone's machine
2. Modifying the docker image to have those plugins + config
3. Figuring out authentication, if any is needed.

Does that help?

[batpad]

Does that help?

Hugely. Thank you!

Modifying the docker image to have those plugins + config

This might be a naive question, but do you have a recommended workflow to test whether "this works"? i.e. run docker image locally some-how and try connecting to the running QGIS or so? Or do we just push changes to the Dockerfile and wait for help to get it deployed to a test environment?

Figuring out authentication, if any is needed.

Would be good to know more about what this might entail? Is this authentication from within QGIS to access the required datasets / catalogs? Or is this some-how managing access to the QGIS setup (which I assume is just handled by the Jupyterhub infrastructure?). We can come back to this if it becomes an issue.

[colliand]

FYI @jmunroe. Discussion here can likely be scooped into improved documentation to enable more community managed customization.

[yuvipanda]

Is this authentication from within QGIS to access the required datasets / catalogs?

Yes, this is what needs to be handled. If it requires AWS creds, it just needs to recognize it's in an environment where it can pick up AWS creds from the env and pass them on. Nothing JupyterHub specific.

This might be a naive question, but do you have a recommended workflow to test whether "this works"? i.e. run docker image locally some-how and try connecting to the running QGIS or so?

You can definitely run it locally! Build the image, run it with:

docker run -it -p 8888:8888 --security-opt seccomp=unconfined <name-of-image>

This should output a bunch of stuff, ending with a url like 'https://127.0.0.1:8888/?token='. You should be able to access jupyterlab with that tokenized URL, and launch qgis from there to see how that goes.

I think to test AWS creds you may need to run in-cluster, but you can also do that easily by building and pushing the image (i recommend a test account on quay.io as registry), and then using the feature of the veda hub that lets you write in the image to use (thanks for helping building that, @batpad ;)). This should reproduce the exact set up so you can see what needs to happen.

[batpad]

cc @Rub21 @sunu - let's figure a time to look at next steps here.

The Earthdata GIS layers are availble via a Rest API and a connection can currently be established manually.

@freitagb is it possible to get a bit more clarity on what this looks like, for those of us with a bit less overall context? Ideal would be a screencast of the manual process, or even just screenshots of having the layers loaded would be great, just so we have a precise idea of the goal / what needs to be replicated in the image definition.

Once we understand the goal correctly / what manual process in QGIS we just need to automate at startup, we can go ahead and experiment with changes to the Docker image to get things to the desired state.

@yuvipanda thanks again for all the super-clear guidance! It feels like there's quite a lot we might want to do with the "QGIS in the browser" feature down the line - figuring out the Earthdata GIS layers sounds like a good first step to get our feet wet, before getting into stuff that could be significantly more complicated, which we can worry about later.

[geohacker]

I was chatting with @batpad about QGIS integration so thought I'll chime in here with some ideas. This is very exciting — I love getting QGIS preloaded with data sources, have them show up as layers etc.

1. Use a QGIS project file (.qgs)

This is probably the least effort method. The steps might look like this — manually add the layers once in QGIS (locally or otherwise), save the QGIS project. Put the project file somewhere the container can read from or pull it while building the image. Write a small startup script that runs when QGIS launches which then loads the project. The project file is quite powerful. They can hold a lot of configuration including default enabled plugins. They can be shared, including any spatial analysis or visualization. Making them almost like notebooks within QGIS!

2. Use a QGIS profile with custom configuration files

QGIS uses a default profile by default. We can change this to have a custom profile which then can have a custom configuration (.conf, .ini, .plist or so) that allows us to edit any QGIS attribute. For example, we can edit projOpenAtLaunchPath variable so QGIS opens the project at launch with no startup scripts. Profiles are really cool because all plugin configuration also lands in the main configuration. For example, the QGIS STAC plugin can take custom STAC connections very easily.

3. A QGIS plugin

A plugin will be really be a nice way to do this but a bit more involved. We can manage the plugin code separately, have qgis configured to load the plugin by default. The plugin can then configure all aspects of working with EGIS data sources including custom layers, styles, managing auth etc. Some folks on our team built plugins extensively (example) that has a lot of UI components, disk reading, network requests, metrics etc.

I'm happy to outline any of these above if helpful.

[batpad]

Thanks for outlining this from the QGIS end @geohacker ! This is all very exciting.

It seems like the quickest way to get the Earthdata GIS Rest API into the hosted QGIS is to just auto-load a custom Project file? This would also allow us to do simple things like ensuring the STAC plugin is activated by default.

It does seem like if we want to start adding more complex features and data sources, the best architecture would be to develop that as a standalone QGIS plugin that we load by default, where we can have a lot of flexibility in what we do.

@j08lue @freitagb we might need a bit of help in getting the exact configuration options to load https://gis.earthdata.nasa.gov/image/rest/services into QGIS, and if any authentication is required, but it seems like then it should be pretty straightforward to create the project file and make the changes required in the container to load that project file on startup, which should give us what we want here.

@geohacker couple more things from your notes that we should follow up on:

They can be shared, including any spatial analysis or visualization. Making them almost like notebooks within QGIS!

Would be great to see if we can build any features to support this?

The plugin can then configure all aspects of working with EGIS data sources

I like the idea of encapsulating all the common data sources and things we know users will want as a separate plugin(s) that can be maintained separately, and useful beyond just QGIS inside JupyterHub. I think how we architect this will depend a bit on if we see this as a one-off feature, or if we envision building more integrations into QGIS in the future. For eg. if we want to allow users to "open dataset in QGIS" from a webpage, we'd definitely need to build it out as a plugin.

Thanks again for the notes @geohacker - it makes it really clear on how we should structure things on the QGIS side and what good short-term and long-term approaches look like.

[geohacker]

Would be great to see if we can build any features to support this?

I've never done this in practice but we have passed around the project file a couple of times and it worked great. You could combine it with version control and then allow people to fork/commit changes. Very speculative, could be possible.

I like the idea of encapsulating all the common data sources and things we know users will want as a separate plugin(s)

For a plugin I might suggest structuring it somewhat as a standalone QGIS server. Then you can easily have triggers and a REST interface to configure, load data, perform other operations etc.

[j08lue]

@j08lue @freitagb we might need a bit of help in getting the exact configuration options to load https://gis.earthdata.nasa.gov/image/rest/services into QGIS, and if any authentication is required

Who would be the right person to connect to, to get these details, @freitagb?

[yuvipanda]

I've moved this image out to its own repo, in https://github.com/2i2c-org/nasa-qgis-image. I've invited @batpad, @freitagb and @slesaad as admins on that repo - feel free to send out access as you wish. It gets automatically built and pushed to https://quay.io/repository/2i2c/nasa-qgis-image?tab=tags

[batpad]

Thanks @yuvipanda !

As I see next steps here:

• Test local setup. See @yuvipanda's comment above
• Modify Docker image / start commands to:
  • Automatically open QGIS in the desktop environment
  • Auto-load a project file that we have pre-loaded into the docker container
• After that or in parallel, we can work on creating a QGIS project file pre-loaded with all the data sources and configuration we need to add the Earthdata Rest APIs and anything else we'd like always pre-loaded by default.

I can start on just figuring out local setup. Once that makes sense, @geohacker @Rub21, we can follow up on next steps to auto-configure QGIS.

[batpad]

I created an issue linked to this issue in the veda-jupyterhub repository: NASA-IMPACT/veda-jupyterhub#9

I narrowed the scope of that issue to creating a project file with the data sources we need. The conversation on this issue is still useful as context, but the veda-jupyterhub repository is likely the more appropriate place to track.

@yuvipanda you can close this if that seems all fine. Thanks!

[yuvipanda]

Going to close this as it's tracked in the veda-jupyterhub issue now. Thanks @batpad