[bug]: Error: ENOENT: no such file or directory, lstat '...' when trying to upload artifact using wildcard on Linux runners with v4.4.1

[Ferroin]

What happened?

Starting with actions/[email protected], something has changed about artifact file selection when using a wildcard which is causing some files in our workflows that are supposed to be included in the artifacts to not be included, with errors being logged essentially claiming those files do not exist. The files in question are all actual files (not symlinks), and downgrading to actions/[email protected] seems to resolve the issue.

Log output from the step doing the upload:

Run actions/upload-artifact@v4
  with:
    name: dist-static-x86_64
    path: artifacts/*.gz.run
    retention-days: 30
    if-no-files-found: warn
    compression-level: 6
    overwrite: false
    include-hidden-files: false
With the provided path, there will be 4 files uploaded
Artifact name is valid!
Root directory input is valid!
Beginning upload of artifact content to blob storage
Warning: ENOENT warning during artifact zip creation. No such file or directory
Error: ENOENT: no such file or directory, lstat 'netdata-x86_64-v1.99.0-287-gb04b15245.gz.run'
Warning: ENOENT warning during artifact zip creation. No such file or directory
Error: ENOENT: no such file or directory, lstat 'netdata-x86_64-v1.99.0-287-gb04b15245.gz.run'
Uploaded bytes 8388608
Uploaded bytes 16777216
Uploaded bytes 25165824
Uploaded bytes 33554432
Uploaded bytes 41943040
Uploaded bytes 50331648
Uploaded bytes 58720256
Uploaded bytes 67108864
Uploaded bytes 75497472
Uploaded bytes 83886080
Uploaded bytes 92274688
Uploaded bytes 100663296
Uploaded bytes 109051904
Uploaded bytes 117440512
Uploaded bytes 125829120
Uploaded bytes 134217728
Uploaded bytes 142606336
Uploaded bytes 150994944
Uploaded bytes 153127480
Finished uploading artifact content to blob storage!
SHA256 hash of uploaded artifact zip is e58b432573661ab658858b898306f7fc23a124bb8521145f687310d4ab9ab4de
Finalizing artifact upload
Artifact dist-static-x86_64.zip successfully finalized. Artifact ID 2030320718
Artifact dist-static-x86_64 has been successfully uploaded! Final size is 153127480 bytes. Artifact ID is 2030320718

What did you expect to happen?

I expected all the files that match the wildcard pattern to actually be found and included.

How can we reproduce it?

Attempt to upload an artifact with files selected using a wildcard pattern using an affected version of actions/upload-artifact. There may be other conditions required to reproduce the issue that I do not know of.

Anything else we need to know?

Despite the similar error message, this is not a duplicate of #240 as it has nothing to do with path-length limitations on Windows runners.

What version of the action are you using?

v4.4.1

What are your runner environments?

linux

Are you on GitHub Enterprise Server? If so, what version?

No response

[robherley]

👋 The bug for symlinks in v4.4.1 will be resolved in v4.4.2 and released to v4 shortly.

The files in question are all actual files (not symlinks)

Can you share file attributes of what you are trying to upload? And can you also try with @v4.4.2?

[Ferroin]

Can you share file attributes of what you are trying to upload?

Not easily unfortunately. All of the files in question though should be ‘regular’ files (no symlinks, directories, or any other file types) owned by the user default user in the GHA runner and with 644 or 755 permissions.

And can you also try with @v4.4.2?

Updating the workflows now to check.

[Ferroin]

Yep, looks like v4.4.2 fixes the issue.

[robherley]

Great!v4 is also updated now so you should be good to go

edit, see: #590 (comment)
v4.4.2 should still be okay to use in the meantime

[Ferroin]

Possibly stupid question, but is there some reason that v4 wasn’t rolled back to v4.4.0 instead, given that that is known working in both the case here and #630? Instead, any workflow that was broken by this bug but got silently fixed by v4 updating to v4.4.2 is now broken again.

[robherley]

Apologies for the back and forth. We decided to rollback to the most recent version that was known working to avoid any further disruptions.

[robherley]

v4.4.3 is released with changes that contain all the fixes, v4 has not been updated yet:

• Unable to upload artifact when using v4.4.2 #630 (comment)

[marc-hb]

Apologies for the back and forth. We decided to rollback to the most recent version that was known working to avoid any further disruptions.

Could you please LOG the upload version number? I landed here because relative symbolic links started to break 2 days ago. The targets of these links exist, always have for years. Last successful daily run: https://github.com/thesofproject/sof/actions/runs/11206675653/job/31147770697

But I'm completely lost in the various issues connected to this and the main reason is because I have absolutely no idea which upload version ran when and where.

First failing run:
https://github.com/thesofproject/sof/actions/runs/11226456767/job/31207017532#step:7:15

Run actions/upload-artifact@v4
  with:
    name: linux-build  mtl
    if-no-files-found: error
    path: /home/runner/work/sof/sof/workspace/build-sof-staging
  /home/runner/work/sof/sof/workspace/**/compile_commands.json
  
    compression-level: 6
    overwrite: false
    include-hidden-files: false
With the provided path, there will be [2](https://github.com/thesofproject/sof/actions/runs/11226456767/job/31207017532#step:7:2)3 files uploaded
Artifact name is valid!
Root directory input is valid!
Beginning upload of artifact content to blob storage
Warning: ENOENT warning during artifact zip creation. No such file or directory
Error: ENOENT: no such file or directory, lstat '../../mtl/community/sof-mtl.ri'
Warning: ENOENT warning during artifact zip creation. No such file or directory
Error: ENOENT: no such file or directory, lstat '../../mtl/community/sof-mtl.ri'

[marc-hb]

The targets of these links exist, always have for years.

Error: ENOENT: no such file or directory, lstat '../../mtl/community/sof-mtl.ri'

Even if the target were missing, broken symbolic links are "valid" Unix objects and should be included in the upload. On my Linux system, zip --symlinks includes both broken and non-broken symlinks in the archive without even a warning.

Anyway the links missing in https://github.com/thesofproject/sof/actions/runs/11226456767/job/31207017532#step:7:15 were not broken.

[markfickett]

I'm trying to use upload-artifact to save/restore my node_modules directory between jobs, and it's not restoring the .bin directory correctly (even with include-hidden-files: true). I see error messages about the symlinks in .bin and I think it may be because the action thinks they're broken (even though they work fine before the upload/download artifact).

Example error:

Warning: ENOENT warning during artifact zip creation. No such file or directory
Error: ENOENT: no such file or directory, lstat '../jest/bin/jest.js'

However, this may just be because if issues with relative symlinks: #590 .

[robherley]

@marc-hb

Could you please LOG the upload version number?

In the "set up job" step of your workflow run, you can see the SHA of the resolved action version:

Download action repository 'actions/upload-artifact@v4' (SHA:b4b15b8c7c6ac21ea08fcf65892d2ee8f75cf882)

You can view that commit to see any tags associated with it, e.g. b4b15b8

Your issue with relative symlinks should be resolved as of v4.4.3 yesterday, and v4 is now updated to include this.

[robherley]

Even if the target were missing, broken symbolic links are "valid" Unix objects and should be included in the upload. On my Linux system, zip --symlinks includes both broken and non-broken symlinks in the archive without even a warning.

Not including broken symlinks have been behavior of this actions since v2 about ~4 years ago, here's the commit/tag it was originally introduced: https://github.com/actions/upload-artifact/blob/v2/src/search.ts#L18

Of course we can always change that, but I'm not sure there's a lot of use cases that require it.

[robherley]

@markfickett Judging from the error, your .bin symlinks look like relative links, and should be fixed now.

[robherley]

Since the fix has been out for about a week now, closing

[marc-hb]

Of course we can always change that [Not including broken symlinks], but I'm not sure there's a lot of use cases that require it.

I think a typical use case for broken symlinks is: "composition". That is: two different archives where one "provides" some symlink targets for the other.

It is clearly an unusual case and there is an obvious and not so inconvenient workaround: pre-tar or pre-zip before the upload. Same workaround as for permissions: https://github.com/actions/upload-artifact?tab=readme-ov-file#permission-loss

Whichever the default is, the ability to pass glob options "through" upload would be really cool! The name getDefaultGlobOptions() sounds like there could be some overriding possibility but I can't tell whether that's possible or not from https://github.com/actions/upload-artifact?tab=readme-ov-file#upload-using-multiple-paths-and-exclusions . It's at least not documented if not implemented.

But I'm afraid the main problem is: SILENTLY dropping broken symbolic links? Consider the case when these links were not supposed to be broken. Now this is not a corner case any more at all but hiding a serious problem from a "regular" user! I didn't test (sorry) but I see https://github.com/actions/toolkit/blob/29d342f1765379a4ab7b028e4c683928f23030a9/packages/glob/src/internal-globber.ts#L207 seems to have only a core.debug("Broken symlink:...")? If that's all there is then that's way too quiet.

My 2 cents, sorry I spent a limited time researching all this.

PS: I found a broken... URL at the bottom of https://github.com/actions/upload-artifact/blob/82c141cc518b40d92cc801eee768e7aafc9c2fa2/README.md

See extra documentation for the @actions/artifact package

https://github.com/actions/toolkit/blob/main/packages/artifact/docs/additional-information.md does not exist.