Path('~/').expanduser() in config validators?

[Peter9192]

I tried setting the path to singularity to ~/ewatercycle/singularity_image.sif, and I found that Pathlib later on failed like so:

Path('~/ewatercycle/singularity_image.sif').exists()  # --> False
Path('home/peter/ewatercycle/singularity_image.sif').exists()  # --> True

I think it would be nice if the config validators automatically resolve the path so we don't have to do it everywhere in the code.

[stefsmeets]

All the images are parsed as strings in the config:

ewatercycle/ewatercycle/config/_validators.py

Line 139 in 9028a37

'wflow.singularity_image': validate_string_or_none,

If they are always paths, could do:
'wflow.singularity_image': validate_path_or_none

That will resolve the ~.

[stefsmeets]

Seems like this does not work, because singularity_image can also be something like: docker://ewatercycle/lisflood-grpc4bmi:latest

[stefsmeets]

Looking at the documentation of grpc4bmi it seems that the image argument must be a string. Not sure if passing a Path object will lead to issues.

[sverhoeven]

From singularity help

  singularity run accepts the following container formats:

  *.sif               Singularity Image Format (SIF). Native to Singularity 3.0+
  
  *.sqsh              SquashFS format.  Native to Singularity 2.4+

  *.img               ext3 format. Native to Singularity versions < 2.4.

  directory/          sandbox format. Directory containing a valid root file 
                      system and optionally Singularity meta-data.

  instance://*        A local running instance of a container. (See the instance
                      command group.)

  library://*         A SIF container hosted on a Library
                      (default https://cloud.sylabs.io/library)

  docker://*          A Docker/OCI container hosted on Docker Hub or another
                      OCI registry.

  shub://*            A container hosted on Singularity Hub.

  oras://*            A SIF container hosted on an OCI registry that supports
                      the OCI Registry As Storage (ORAS) specification.

I like docker://* as it allows me to skip creating a sif file.

[Peter9192]

I also like the docker://*, but does that mean the image will be built "just in time", every time? Or can it use cached images if it finds them? Since creating the sif file can take quite long (20 minutes or so?)

[sverhoeven]

Yep, first time its downloaded, converted into Singularity format and stored in ~/.singularity/cache. After that it will use cached image.

[Peter9192]

Also see eWaterCycle/grpc4bmi#100

[Peter9192]

Closing in favour of #119 and #93