Rewamp Docker file for home archiving

[danielhollas]

Build on top of #764
Alternative to #740

The main goal here is to reduce the complexity of status quo and of #740.

• The strategy of archiving home directory and untarring it at startup actually allows for a lot of simplification of the Dockerfile since everything can be directly prepared in home folder, without intermediary steps, and this allows to get rid of the current startup scripts (70_, 71_)
• All startup scripts from full-stack are preserved and reused, which minimizes duplication, resolves the SSH key issue and should be more maintainable
• The only new startup script is the 00_untar_home.sh which is basically the same here as in Docker image with home directory tar #740.
• I've also done some unrelated cleanup:
  • Moved Dockerfile to the root dir, and got rid of build.json and docker-bake.hcl which were introducing unnecessary complexity. One can now build the image by simply running docker build . -t aiidalab/qe:newbuild . This also allowed me to use podman on my dev machine, which otherwise doesn't support the buildx backend.

I've done some quick benchmarking, at starting the container takes around 12s on my machine. The image takes around 6.5Gb. We could trade around 300Mb images size for extra 3s of startup time if we compressed the home.tar archive. (My timings seems roughly consistent with those observed in #740.

@unkcpz @superstar54 Let me know what you think. I'd recommend to ignore the noise in the diff and just look at the new Dockerfile to see if that makes sense.

I am happy to split this up to smaller pieces if somebody will actually review them. :-P

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 68.28%. Comparing base (2c201f0) to head (a493711).
Report is 46 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #773   +/-   ##
=======================================
  Coverage   68.28%   68.28%           
=======================================
  Files          45       45           
  Lines        4143     4143           
=======================================
  Hits         2829     2829           
  Misses       1314     1314           

Flag	Coverage Δ
python-3.10	?
python-3.11	68.28% <ø> (?)
python-3.9	68.31% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[danielhollas]

Since we're copying the full contents of the repo in the image, I don't think we can do any filtering here...

[unkcpz]

Thanks @danielhollas. Just go through the PR, some minor comments, let's discuss tomorrow.

[unkcpz]

This is a bit confusing can you explain what it achieved? Is it equivalent as original one?

[danielhollas]

Yeah, I can add a comment, this is explained more in #764 on which this PR builds. Could you please take a look and review that separately? This just means that tests with aiida=2.5 will not run with --cov which speeds them up quite a lot.

[unkcpz]

Nice stuff!

[unkcpz]

I think this check exist in @superstar54's implementation, but why it is necessary, when is the home can be empty?

[danielhollas]

I just wanted to guard against if somebody executed this script by mistake outside of the container.

[danielhollas]

Thanks @danielhollas. Just go through the PR, some minor comments, let's discuss tomorrow.

Thanks! It would be great if you at @superstar54 could give it a test, run the container and open the QeApp and try running some calculation to see if things are working, it's possible I overlooked something.

[unkcpz]

run the container and open the QeApp and try running some calculation to see if things are working, it's possible I overlooked something.

The PR to ghcr is not triggered, if you can bring it back, I think it is even better to test on the Azure k8s.

[superstar54]

Hi @danielhollas , thanks for the great work. I have tested locally, and the QEApp is running fine.

One comment: in #740 , it only takes 6 s to launch, and this PR needs around ~ 12 s, any improvement can be done?

[danielhollas]

I don't know, feel free to investigate. But I suspect that in #774 you're not doing some work that actually needs to get done, e.g. generating the SSH key and running database migration. Those two things easily add 3s.

[danielhollas]

You can test how much time it takes if you run it for the second time (with the volume attached). If there is any time difference, that would suggest that an improvement is possible

[danielhollas]

Closing in favour of #778