security.devlxd.images doesn't work

[simondeziel]

Bug discovery

For some reason, pulling images:alpine/edge was pathologically slow for me/today (~15kB/s):

root@v2:~# lxc launch images:alpine/edge a1
Creating a1
Retrieving image: rootfs: 15% (14.84kB/s)   
Retrieving image: rootfs: 20% (14.39kB/s)   
Retrieving image: rootfs: 60% (14.45kB/s)   
Retrieving image: rootfs: 90% (15.09kB/s)   
Starting a1

root@v2:~# lxc image ls
+-------+--------------+--------+-----------------------------------+--------------+-----------+---------+-----------------------------+
| ALIAS | FINGERPRINT  | PUBLIC |            DESCRIPTION            | ARCHITECTURE |   TYPE    |  SIZE   |         UPLOAD DATE         |
+-------+--------------+--------+-----------------------------------+--------------+-----------+---------+-----------------------------+
|       | 3fa9cc8313d7 | no     | Alpine edge amd64 (20241205_0022) | x86_64       | CONTAINER | 3.09MiB | Dec 5, 2024 at 6:11pm (UTC) |
+-------+--------------+--------+-----------------------------------+--------------+-----------+---------+-----------------------------+

Yet, that image is already in my host's cache:

sdeziel@sdeziel-lemur:~$ lxc image ls | grep -wF 3fa9cc8313d7
| alpine-edge | 3fa9cc8313d7 | no     | Alpine edge amd64 (20241205_0022)                 | x86_64       | CONTAINER       | 3.09MiB   | Dec 5, 2024 at 5:59pm (UTC)  |

Simple reproducer

In a first terminal

$ lxc launch ubuntu-minimal-daily:24.04 c2 -c security.devlxd.images=true
Launching c2
$ lxc exec c2 -- snap install lxd --channel latest/edge
2024-12-05T18:36:58Z INFO Waiting for automatic snapd restart...
lxd (edge) git-ac7aa74 from Canonical✓ installed
$ lxc exec c2 -- lxd init --auto
$ sleep 5  # allow time for LXD to do a bunch of network requests

# Using shell here is important (rather than another exec command)
$ lxc shell c2

In another terminal:

# images.lxd.canonical.com IPs
sudo tcpdump -ni any host 185.125.189.121 or host 2620:2d:4000:100c::147

In yet another terminal:

lxc monitor --pretty

Then back to the first terminal that is in c2:

root@c2:~# lxc image copy images:alpine/edge local:
Image copied successfully!                   
root@c2:~# lxc image ls
+-------+--------------+--------+-----------------------------------+--------------+-----------+---------+-----------------------------+
| ALIAS | FINGERPRINT  | PUBLIC |            DESCRIPTION            | ARCHITECTURE |   TYPE    |  SIZE   |         UPLOAD DATE         |
+-------+--------------+--------+-----------------------------------+--------------+-----------+---------+-----------------------------+
|       | 3fa9cc8313d7 | no     | Alpine edge amd64 (20241205_0022) | x86_64       | CONTAINER | 3.09MiB | Dec 5, 2024 at 6:43pm (UTC) |
+-------+--------------+--------+-----------------------------------+--------------+-----------+---------+-----------------------------+

That lxc image copy will do network traffic (visible in the tcpdump terminal) and nothing in the lxc monitor one. Note the exact same fingerprint was retried.

Additional information

$ snap list lxd
Name  Version      Rev    Tracking     Publisher   Notes
lxd   git-ac7aa74  31567  latest/edge  canonical✓  -

[markylaing]

I suspect this is a case where the client doesn't check if the devlxd socket is present on this particular code path. I've never had an issue with it on lxc launch/init. So perhaps when copying an image it doesn't perform the same checks.

[tomponline]

I've marked this as an improvement, as its not clear whether it was ever intended for devlxd images endpoint to work with image copy. But it seems like something worth doing if feasible.

[simondeziel]

To be clear, I noticed the issue with a lxc launch but resorted to lxc image copy as the simplest case I could think of for a reproducer.

[markylaing]

It seems that the reason for this is that not all images are being marked as cached:

$ lxc image list
+-------+--------------+--------+---------------------------------------------+--------------+-----------------+-----------+------------------------------+
| ALIAS | FINGERPRINT  | PUBLIC |                 DESCRIPTION                 | ARCHITECTURE |      TYPE       |   SIZE    |         UPLOAD DATE          |
+-------+--------------+--------+---------------------------------------------+--------------+-----------------+-----------+------------------------------+
|       | 40d8df642812 | no     | ubuntu 24.04 LTS amd64 (release) (20241119) | x86_64       | CONTAINER       | 254.20MiB | Dec 18, 2024 at 3:41pm (UTC) |
+-------+--------------+--------+---------------------------------------------+--------------+-----------------+-----------+------------------------------+
|       | 42ac4cf1ed33 | no     | ubuntu 22.04 LTS amd64 (release) (20241217) | x86_64       | CONTAINER       | 424.91MiB | Dec 19, 2024 at 9:46am (UTC) |
+-------+--------------+--------+---------------------------------------------+--------------+-----------------+-----------+------------------------------+
|       | cac3939319e8 | no     | ubuntu 24.04 LTS amd64 (release) (20241119) | x86_64       | VIRTUAL-MACHINE | 579.23MiB | Dec 19, 2024 at 9:40am (UTC) |
+-------+--------------+--------+---------------------------------------------+--------------+-----------------+-----------+------------------------------+
|       | cb37cdd2ed1b | no     | Alpine edge amd64 (20241219_0022)           | x86_64       | CONTAINER       | 3.09MiB   | Dec 19, 2024 at 9:42am (UTC) |
+-------+--------------+--------+---------------------------------------------+--------------+-----------------+-----------+------------------------------+
|       | e892f376d856 | no     | ubuntu 22.04 LTS amd64 (release) (20241217) | x86_64       | VIRTUAL-MACHINE | 604.57MiB | Dec 18, 2024 at 4:16pm (UTC) |
+-------+--------------+--------+---------------------------------------------+--------------+-----------------+-----------+------------------------------+
$ lxd sql global 'SELECT fingerprint, filename, public, cached, type FROM images'
+------------------------------------------------------------------+-----------------------------------------------+--------+--------+------+
|                           fingerprint                            |                   filename                    | public | cached | type |
+------------------------------------------------------------------+-----------------------------------------------+--------+--------+------+
| 40d8df642812cf14fceb8db901f30b415d4829e79b82bbccf7dc73f94f0205f6 | ubuntu-24.04-server-cloudimg-amd64-lxd.tar.xz | 0      | 1      | 0    |
| e892f376d85619000fd80d3f7adced6c72aac9e94f53cdc0dd6ab1fd0c5400a9 | ubuntu-22.04-server-cloudimg-amd64-lxd.tar.xz | 0      | 1      | 1    |
| 40d8df642812cf14fceb8db901f30b415d4829e79b82bbccf7dc73f94f0205f6 |                                               | 0      | 0      | 0    |
| cac3939319e81e626a8a9daf564a6285b6eb5240882e557f5203893bd58285c8 | ubuntu-24.04-server-cloudimg-amd64-lxd.tar.xz | 0      | 1      | 1    |
| cb37cdd2ed1b2daabaa56105abfb49460455463c7d9db3d0ed8ccef5efeb0ba6 |                                               | 0      | 0      | 0    |
| 42ac4cf1ed33cb2de4c2582252786bd15f5bbeea79436af4ed010b81b2b45fb5 |                                               | 0      | 0      | 0    |
+------------------------------------------------------------------+-----------------------------------------------+--------+--------+------+

I'll try and figure out the reason for this.

[markylaing]

I noticed this because I found that it was working fine for ubuntu:24.04 as in our CI, but not for alpine