Adding an ostree commit to an existing repository with a new ref makes the old ref unavailable

[achilleas-k]

Version
pulpcore version: 3.30.0
pulp-ostree version: 2.1.1

I'm using the single container docker.io/pulp/pulp:latest (image ID: 5f86bd6a6a4197256d0786509bc3f0072b177db4db517cd0388903dc8d7dc73d).

Describe the bug
When two ostree commits with different refs are are added to the same repository, the old ref disappears. I can see both refs and commits when I browse the repository files in the browser, but the summary file only shows the one I pushed second (running ostree remote summary <remotename> with the remote configured). There's no way to pull the older ref.

To Reproduce
Build two ostree commits with two different refs. In my case it was fedora/38/x86_64/iot and fedora/39/x86_64/iot (built with image builder).

Upload the first commit, create a new repository, and distribute it (for this example, the repo URL will be http://localhost:8080/pulp/content/test/)

Create a local repository and set up the remote:

$ mkdir repo

$ ostree init --repo ./repo

$ cd repo

$ ostree remote add --no-gpg-verify iot http://localhost:8080/pulp/content/test/

$ ostree remote refs iot
iot:fedora/38/x86_64/iot

$ ostree remote summary iot
* fedora/38/x86_64/iot
    Latest Commit (18.9 kB):
      1598deb49e91cc1fb38048bb6e5d62a70174d8ceaf461a9b3f1669e9e809c181
    Version (ostree.commit.version): 38
    Timestamp (ostree.commit.timestamp): 2023-08-16T19:18:28+02

Last-Modified (ostree.summary.last-modified): 2023-08-16T21:51:06+02
Repository Mode (ostree.summary.mode): archive-z2
Has Tombstone Commits (ostree.summary.tombstone-commits): No
ostree.summary.indexed-deltas: true

Upload the second commit, with ref fedora/39/x86_64/iot and add it to the same repository.
Then, once the import task is done, check the remote again:

$ ostree remote refs iot
iot:fedora/39/x86_64/iot

$ ostree remote summary iot
* fedora/39/x86_64/iot
    Latest Commit (15.0 kB):
      350f23c2584373924b0e5ab95e55f135fb11508961379d1730e7212e7e5a1320
    Version (ostree.commit.version): 39
    Timestamp (ostree.commit.timestamp): 2023-08-10T12:07:01+02

Last-Modified (ostree.summary.last-modified): 2023-08-16T22:06:45+02
Repository Mode (ostree.summary.mode): archive-z2
Has Tombstone Commits (ostree.summary.tombstone-commits): No
ostree.summary.indexed-deltas: true

Trying to pull the fedora 38 commit fails:

$ sudo ostree pull iot fedora/38/x86_64/iot
error: No such branch 'fedora/38/x86_64/iot' in repository summary

To verify, both refs are available in the repo:

$ curl http://localhost:8080/pulp/content/test/refs/heads/fedora/38/x86_64/iot
1598deb49e91cc1fb38048bb6e5d62a70174d8ceaf461a9b3f1669e9e809c181


$ curl http://localhost:8080/pulp/content/test/refs/heads/fedora/39/x86_64/iot
350f23c2584373924b0e5ab95e55f135fb11508961379d1730e7212e7e5a1320

Expected behavior
Both refs should appear in the repository summary and be available to pull.

Additional context
To upload the commit tarball, create the repository, import the commit, and distribute the repo, I'm using an autogenerated client library: https://github.com/osbuild/pulp-client
The goal is to be able to push ostree commits built by image builder (osbuild-composer) directly into a repository hosted by Pulp.

[lubosmj]

Can I get a complete reproducer or a script that led to this behaviour? I tried the following and everything worked:

# prepare pulp
pulp ostree repository create --name test
pulp ostree distribution create --name test --repository test --base-path test

# first commit
ostree --repo=repo init --mode=archive
mkdir files
echo "asdasd" > files/file.txt
COMMIT=$(ostree commit --repo repo --branch foo files/)
tar czvf repo.tar repo/

# first upload
pulp ostree repository import-all --name test --file repo.tar --repository_name repo

# local remote repo
ostree --repo=rremote init
ostree --repo=rremote remote add --no-gpg-verify iot http://localhost:5001/pulp/content/test/
ostree --repo=rremote remote refs iot
ostree --repo=rremote remote summary iot

# second commit
echo "asdasd" > files/file2.txt
ostree commit --repo repo --branch boo files/ --parent=$COMMIT
tar czvf repo2.tar repo/

# second upload
pulp ostree repository import-all --name test --file repo2.tar --repository_name repo

# fetch refs
ostree --repo=rremote remote refs iot
ostree --repo=rremote remote summary iot
ostree --repo=rremote pull iot foo
ostree --repo=rremote pull iot boo

[achilleas-k]

Here you go.

#!/usr/bin/env bash

set -euo pipefail

workdir=$(mktemp -d)
cd "${workdir}"

# prepare pulp
pulp ostree repository create --name test
pulp ostree distribution create --name test --repository test --base-path test

# first commit
mkdir "${workdir}/first"
cd "${workdir}/first"
ostree --repo="${workdir}/first/repo" init --mode=archive
mkdir "${workdir}/first/files"
echo "one" > files/file.txt
ostree commit --repo "${workdir}/first/repo" --branch first "${workdir}/first/files/"

cd "${workdir}/first"
tar czvf repo.tar "repo/"

# first upload
pulp ostree repository import-all --name test --file repo.tar --repository_name repo

# local remote repo
ostree --repo="${workdir}/rremote" init
ostree --repo="${workdir}/rremote" remote add --no-gpg-verify iot http://localhost:8080/pulp/content/test/
ostree --repo="${workdir}/rremote" remote refs iot
ostree --repo="${workdir}/rremote" remote summary iot

# second commit
mkdir "${workdir}/second"
cd "${workdir}/second"
mkdir files
echo "two" > files/file2.txt
ostree --repo="${workdir}/second/repo" init --mode=archive
ostree commit --repo repo --branch second files/
tar czvf repo.tar repo/

# second upload
pulp ostree repository import-all --name test --file repo.tar --repository_name repo

# fetch refs
ostree --repo="${workdir}/rremote" remote refs iot
ostree --repo="${workdir}/rremote" remote summary iot
ostree --repo="${workdir}/rremote" pull iot second
ostree --repo="${workdir}/rremote" pull iot first  # error: No such branch 'first' in repository summary

Note main difference: The two commits are created in completely separate repositories and don't share a parent-child relationship.

For comparison, this is a similar scenario with the two separate repos with two unrelated commits being pulled into a local repository with pull-local:

#!/usr/bin/env bash

set -euo pipefail

workdir=$(mktemp -d)
cd "${workdir}"

# first commit
mkdir "${workdir}/first"
cd "${workdir}/first"
ostree --repo="${workdir}/first/repo" init --mode=archive
mkdir "${workdir}/first/files"
echo "one" > files/file.txt
ostree commit --repo "${workdir}/first/repo" --branch first "${workdir}/first/files/"

# local pull repo
ostree --repo="${workdir}/rremote" init
ostree --repo="${workdir}/rremote" pull-local "${workdir}/first/repo"
ostree --repo="${workdir}/rremote" summary -u
ostree --repo="${workdir}/rremote" summary -v
ostree --repo="${workdir}/rremote" refs

# second commit
mkdir "${workdir}/second"
cd "${workdir}/second"
mkdir files
echo "two" > files/file2.txt
ostree --repo="${workdir}/second/repo" init --mode=archive
ostree commit --repo repo --branch second files/

# fetch refs
ostree --repo="${workdir}/rremote" pull-local "${workdir}/second/repo"
ostree --repo="${workdir}/rremote" summary -u
ostree --repo="${workdir}/rremote" summary -v
ostree --repo="${workdir}/rremote" refs
ostree --repo="${workdir}/rremote" log first
ostree --repo="${workdir}/rremote" log second

[lubosmj]

If those commits were created in separate repositories locally, why did not we upload them to separate repositories in Pulp? I am having a hard time understanding the actual user workflow here.

I need to make Pulp more robust to "merge" unrelated refs:

pulp_ostree/pulp_ostree/app/tasks/importing.py

Line 271 in 3c93b73

self.repo.regenerate_summary()

[achilleas-k]

This workflow emulates what we do in osbuild when building a commit. The composition of the child commit happens in a separate repository but it is created as a child, using --parent (which is a valid way to build ostree commits). As for pushing them to the same repo, a common workflow should be for users to build upgrade commits and keep pushing them to the same repository which they have statically configured on their clients to pull from.

[achilleas-k]

This workflow emulates what we do in osbuild when building a commit. The composition of the child commit happens in a separate repository but it is created as a child, using --parent (which is a valid way to build ostree commits). As for pushing them to the same repo, a common workflow should be for users to build upgrade commits and keep pushing them to the same repository which they have statically configured on their clients to pull from.

Reviewing this issue issue thread now that it's closed I realise that my last comment might be confusing or misleading. It's possible I confused this issue with #279. The issue here was that a single repository should be able to hold multiple refs (branches), even ones with unrelated histories.

Given the description of the fix, I'm sure it's all fine. I just wanted to clarify the comment in case it creates any confusion in the future.