Add Pynio to image - take 2

[chiaral]

I have been trying to install it locally (conda install -c conda-forge pynio) with no luck.

I know about cfgrib as an alternative engine to read grib files, but cfgrib does not like my grib files - pynio does (I tested it on another machine).

[pangeo-bot]

/condalock
Automatically locking new conda environment, building, and testing images...

[chiaral]

When I try to install it locally on my cluster I get these issues - so I want to raise them before being added to the docker image.
I remember that in the past pynio installation was a bit problematic. let me know!

(notebook) jovyan@jupyter-chiaral:~$ conda install -c conda-forge pynio
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: | 
Found conflicts! Looking for incompatible packages.
This can take several minutes.  Press CTRL-C to abort.
Examining conflict for docker-pycreds tifffile nbstripout pyasn1-modules bleach gsw pytools cached-property color- a

and it's still spinning

[scottyhq]

cross-posting pangeo-data/pangeo-cloud-federation#918 (comment)

FYI - the pangeo-docker-images are shared on all hubs (GCP too), and the fact that conda install pynio is hanging to find a solution makes me worried it has complicated dependencies. in case incorporating into the shared docker images does not work, you can try: mamba install pynio . I've noticed mamba succeeds where conda fails for adding some of these complex geospatial libraries into our existing environment.

And if you need pynio on dask workers this is the current workaround available:
pangeo-data/pangeo#792 (comment)

[scottyhq]

i'm worried that pynio is causing many other packages to drop major or minor versions (also gdal, cartopy, tiledb, scipy, etc...)

[chiaral]

yeah - maybe it's better to not merge it - i can try something else for now.

[rabernat]

Thanks @scottyhq for trying to help sort this out!

I think this issue highlights the need for users to be able to bring their own image to Pangeo cloud deployments. It would be great if @chiaral could build her own docker image with pynio in it and select it at startup time. I'll ping @yuvipanda on this as an example of one of the features we hope to have in 2i2c world.

[chiaral]

Thanks! I understand the issue - and indeed pynio has always been a bit of a wild card.
I will try mamba for sure. So do not merge it to master just yet.

[scottyhq]

@chiaral also, just updated my comment above (#182 (comment)) with a link to instructions for getting pynio on dask workers if you need that

It would be great if @chiaral could build her own docker image with pynio in it and select it at startup time. I'll ping @yuvipanda on this as an example of one of the features we hope to have in 2i2c world.

Agreed, pointing to another user-defined image would be great. A similar approach that i haven't had time to explore is letting users swap out the conda environment layer in the existing docker image from a hub:
#148

[chiaral]

Thanks - I used mamba and it worked right away - and indeed downgraded tons of stuff.
For now I am exploring without dask workers - for this - so I won't need the workers, but I will keep that in mind. thanks!

[chiaral]

I am commenting on this issue because I would need some guidance. I used to be able to install in my notebook pynio with mamba no problem, but now I get this error:

Looking for: ['pynio']

conda-forge/noarch       [====================] (00m:01s) Done
conda-forge/linux-64     [====================] (00m:04s) Done
Encountered problems while solving.
Problem: nothing provides libgdal 2.1.* needed by pynio-1.5.0-py27_0

There hasn't been a new release in pynio, so of course it's a new imcompatibility with the rest of the image that has been updated since I opened this issue.
Do you have any suggestion on how to go about this?

I could go around pynio with cfgrib by doing some hacking, and I will happily do that (see this issue ) but i wasn wondering if anyone has any input.
I definitely don't want to break the current docker image.

[scottyhq]

closing this as it seems pynio needs updating to integrate into the default environment