squash refs/convert/... branch history after each series of commits

[severo]

fixes #1774

[severo]

Sorry to bother @Wauplin: do you know why we cannot squash the refs/convert/{parquet,duckdb} branches yet on hub-ci? (see errors in the CI)

[Wauplin]

do you know why we cannot squash the refs/convert/{parquet,duckdb} branches yet on hub-ci?

cc @Pierrci @coyotte508 for that (dunno myself)

What's sure is that squashing normal branches works correctly on hub-ci

[coyotte508]

cc @Kakulukian it should be lmerged: https://github.com/huggingface-internal/moon-landing/pull/12253

[Kakulukian]

API works fine on hub-ci maybe refs/convert/parquet needs to be urlencoded maybe not supported with the current method of hfh

[lhoestq]

lgtm ! feel free to merge once it's all good for you

[severo]

I'll try to pass the tests, otherwise we will have the same issue in prod without a doubt.

[coyotte508]

API works fine on hub-ci maybe refs/convert/parquet needs to be urlencoded maybe not supported with the current method of hfh

cc @hanouticelina maybe

[severo]

The error is:

E           huggingface_hub.errors.HfHubHTTPError: 404 Client Error: Not Found for url: https://hub-ci.huggingface.co/api/datasets/DVUser/csv-17379738584545/super-squash/refs/convert/parquet (Request ID: Root=1-67976585-22683a2a39a5eeb41f7b288a;94d07c7c-ce88-40f3-994b-239988927311)

https://github.com/huggingface/dataset-viewer/actions/runs/12911690116/job/36214768824?pr=3131#step:8:9086

[Wauplin]

(...)csv-17379738584545/super-squash/refs/convert/parquet 

=> indeed a url-encoding issue yes. Will fix that

[severo]

https://github.com/huggingface/huggingface_hub/blob/main/src/huggingface_hub/hf_api.py#L3915

vs

https://github.com/huggingface/huggingface_hub/blob/1ba1b1da4db9db253c24859e9c2f82a61b04ea44/src/huggingface_hub/hf_api.py#L3344

[severo]

hmmm it was not an issue :)

[severo]

let's wait for the next hfh release, upgrade here, and we should be good

[Wauplin]

Yep, sorry about the closing 😬

huggingface_hub 0.28.0 is out! https://github.com/huggingface/huggingface_hub/releases/tag/v0.28.0

[severo]

yes, thanks! I'm upgrading the deps

[severo]

Something strange, maybe related to #3092, @lhoestq while upgrading in webhook service. Could you have a special look at this part?

➜  webhook git:(squash_references) ✗ poetry update huggingface-hub libcommon libapi
Updating dependencies
Resolving dependencies... (44.1s)

Package operations: 1 install, 1 update, 0 removals

  - Installing pyarrow-hotfix (0.6)
  - Downgrading datasets (3.0.3.dev0 1946182 -> 2.19.1 1946182)

Writing lock file

Plus: the operation in the git diff is the opposite:

-version = "2.19.1"
+version = "3.0.3.dev0"

[severo]

OK, a lot of type errors, related to huggingface/huggingface_hub#2752 I guess. Not very important, but let's try to fix them here too

[severo]

Still failing. I made it work when using the repository user token, but it fails when using the bot user token. So: it's an issue in moon-landing: the bot should be able to squash refs/convert/... branches. I opened an issue: https://github.com/huggingface-internal/moon-landing/issues/12355

cc @coyotte508 @Kakulukian @Pierrci FYI (I could open a PR, but I don't feel super confident for auth-related stuff)

[coyotte508]

PR open

[severo]

OK, the squash works. But we have another issue, possibly due to the hfh upgrade

[Wauplin]

Happy to investigate if it's hfh-related but looks like a credential issue to me:

E               huggingface_hub.errors.RepositoryNotFoundError: 401 Client Error. (Request ID: Root=1-679cfee5-239b138677e055ab65d1ee0e;f1978d0d-1f12-422d-9c57-f73597e6a3e5)
E               
E               Repository Not Found for url: https://huggingface.co/api/datasets/severo/winogavil.
E               Please make sure you specified the correct `repo_id` and `repo_type`.
E               If you are trying to access a private or gated repo, make sure you are authenticated.
E               Invalid credentials in Authorization header

[coyotte508]

note that it needs to be an app token associated to a user (eg like the app token that pushes the commits to the parquet ref currently)

[severo]

OK, the 401 error was already present in a previous commit: https://github.com/huggingface/dataset-viewer/actions/runs/12828524481/job/36584381335. So... I think we can merge this one and then fix the issue with 401.

Can it be linked with https://huggingface.slack.com/archives/C02EMARJ65P/p1738582122666929?thread_ts=1672852567.996239&cid=C02EMARJ65P (internal)?

cc @lhoestq wdyt

[lhoestq]

Sure feel free to merge already, it should work directly once the issue is fixed