Pulp file sync for remote manifest files

[pranavkumar74980]

I am unable to sync repo for manifest file (example-> https://example.com/xyz.yaml ) present over remote url.
However, I am able to upload content on pulp repo for locally present yaml manifest files. Is there any way to do to sync remotely without downloading on localhost ? I am unable to find such things in document.
Repo details

pulp file repository list
[
{
"pulp_href": "/pulp/api/v3/repositories/file/file/01939b23-e5e2-726c-9bc0-8f523e847cf5/",
"prn": "prn:file.filerepository:01939b23-e5e2-726c-9bc0-8f523e847cf5",
"pulp_created": "2024-12-06T08:44:01.380551Z",
"pulp_last_updated": "2024-12-06T08:44:01.404199Z",
"versions_href": "/pulp/api/v3/repositories/file/file/01939b23-e5e2-726c-9bc0-8f523e847cf5/versions/",
"pulp_labels": {},
"latest_version_href": "/pulp/api/v3/repositories/file/file/01939b23-e5e2-726c-9bc0-8f523e847cf5/versions/0/",
"name": "mpi-operator-repo",
"description": null,
"retain_repo_versions": null,
"remote": null,
"autopublish": false,
"manifest": "PULP_MANIFEST"
}
]

Below errors:
pulp file repository sync --name mpi-operator-repo --remote mpi-operator-remote
Started background task /pulp/api/v3/tasks/01939b24-4000-730d-825e-60b9fdc335cd/
Error: Task /pulp/api/v3/tasks/01939b24-4000-730d-825e-60b9fdc335cd/ failed: 'Error: Parsing of the manifest file failed on line:9.
Please make sure the remote URL is pointing to a valid manifest file.
The manifest file should be composed of lines in the following format: <relative_path>,,.'

[gerrod3]

Can you post what the contents of the manifest file are? Like the error message says the sync is expecting a specific format for each line so I would first check that your file follows the format correctly.

I am unable to sync repo for manifest file (example-> https://example.com/xyz.yaml ) present over remote url. However, I am able to upload content on pulp repo for locally present yaml manifest files. Is there any way to do to sync remotely without downloading on localhost ? I am unable to find such things in document. Repo details

I am not sure I understand what your question means. You can always do an on_demand sync by setting the policy on the remote, but syncing still requires downloading and parsing the manifest file to find the artifacts to sync.

[pranavkumar74980]

I ran these command:
pulp file remote create --name k8s-manifest-remote --url https://raw.githubusercontent.com/kubeflow/mpi-operator/v0.4.0/deploy/v2beta1/mpi-operator.yaml --policy on_demand

pulp file repository create --name k8s-manifest-repo

Once I try to sync this I get below errors:
pulp file repository sync --name k8s-manifest-repo --remote k8s-manifest-remote

However, my second paragraph means I was able to upload after downloading it on my machine using
"pulp file content upload" command

Is there any way to sync above one without downloading it on machine?

[pranavkumar74980]

@gerrod3 Is there any missing part or it is something under consideration?

[gerrod3]

I think I understand what you are missing. You are trying to "sync" these yml files into a pulp-file repo without downloading & uploading them manually, correct? In order to sync files they need to be present in a format that pulp-file can understand. Take a look at an example repository we use in our tests to understand the repository layout pulp-file expects to find: https://fixtures.pulpproject.org/file/. The PULP_MANIFEST file is the metadata file listing out all the available files to be synced, so if you wanted to sync these three files you would create a remote like so:
pulp file remote create --name file-fixtures --url https://fixtures.pulpproject.org/file/PULP_MANIFEST.

If there is no easy way to group all the yml files you want into a repository that you can sync, you could create the content one by one using the file-url field on file create. e.g.
pulp file content create --relative-path $FILENAME --file-url $FILE_URL --repository $REPO_NAME.
This will have Pulp go and download the file at the url and create content from it and then add it to your repository.