Support FAT file attributes on non-Windows platforms

[Thealexbarney]

.NET Core currently does not support getting or setting many file attributes in FAT filesystems on non-Windows platforms..

There is a syscall in Linux that allow reading and writing FAT attributes. As these attributes are filesystem-specific, they could be ignored for filesystems that don't support them.

Having this feature would be useful for ensuring consistent behavior between platforms.

[JeremyKuhne]

I have no problems with this conceptually if someone wants to try to prototype an implementation.

[Thealexbarney]

This is my first time looking at the CoreFX codebase or doing Linux interop, but I tried creating a basic implementation.

Thoughts:
This involves a few changes in existing behavior, such as the FAT hidden flag. On a FAT filesystem in Linux, files beginning with a . will show up as hidden, but they won't on Windows.
The readonly flag is not an issue because FAT filesystems didn't have standard Unix permissions to begin with.
I'm not sure what to do with the attribute logic in FileSystemEntry.
Will this have any performance impact?
Some of the functions used operate on strings instead of spans, so there's a ToString required. That should probably be changed.
I don't know if macOS supports this.

[JeremyKuhne]

I don't know if macOS supports this.

We check for API availability in configure.cmake. And use the defines created there in the code.

files beginning with a . will show up as hidden

My gut feel is that we should report hidden on Unix whenever there is a leading period, even if the file system has a hidden attribute as well. We did the same thing for Mac I believe.

Will this have any performance impact?

It would, and we'd have to measure it. And we'd want to look at the impact of checking the filesystem before making these calls. If it's super costly we can consider making it an opt-in feature.

Some of the functions used operate on strings instead of spans

That's fine, but we need to be super careful that we're always null terminated.

[Thealexbarney]

Did some performance testing and the results aren't great.

Iterating through filesystem entries takes about 2us each on my machine. Simply doing an open/close on an entry takes a little over 4us.

Enumerating entries and reading FAT attributes on every one takes 3x as long as not reading FAT attributes.

[Thealexbarney]

Since the performance impact is nowhere near negligible, what would making the feature opt-in look like?

[JeremyKuhne]

Since the performance impact is nowhere near negligible, what would making the feature opt-in look like?

Probably adding another flag to EnumerationOptions. At least that is what I would propose as a starting point.

[Thealexbarney]

Probably adding another flag to EnumerationOptions. At least that is what I would propose as a starting point.

That would still leave APIs like File.GetAttributes() and FileSystemInfo. Making it opt-in there wouldn't be as unintrusive as adding a flag to EnumerationOptions.

Aside from having it opt-in at some higher level, which seems like a poor option, the obvious options I see would be adding new overloads to the methods and constructors.

[carlossanlop]

Triage:
Would you be interested in writing the new API proposal, @Thealexbarney ?
See https://github.com/dotnet/runtime/blob/master/docs/project/api-review-process.md

[Thealexbarney]

I'd be interested, @carlossanlop. The issue is how to do it without a performance regression on Linux. A new flag for EnumerationOptions could work for the file enumeration APIs, leaving at least the FileSystemInfo class and File.GetAttributes() to deal with. If the performance hit is still an issue there, adding a new API would work around that, but that leaves a difference in functionality between runtimes as a "gotcha" that people might run across.

[JeremyKuhne]

but that leaves a difference in functionality between runtimes

New functionality in Core that doesn't exist in Framework is normal and expected. We absolutely don't want to regress performance in the existing APIs.

There are tricky aspects to making an API for this that will impact getting it approved, notably what do we do about other file systems? NTFS on Unix? etc...

1. Having separate flags for every type isn't terribly usable or plausible to add to an existing options enum
2. Lumping everything into a single flag would have performance and behavior implications (what if you only care about one/some, what happens when we add additional support)

I think this will be a good exercise to go through. Investigating what it would take to add a second file system would be important I think. One important caveat: I am not familiar with the APIs around this space so there may be other major considerations I'm not aware of yet. :)

[Thealexbarney]

New functionality in Core that doesn't exist in Framework is normal and expected.

I was thinking more of functionality differences between Windows and non-Windows platforms. Like where the standard File.GetAttributes() would get FAT attributes on Windows, but a different method would be used on Linux.

As for what a set of extensible file system APIs might look like, here's what first comes to mind.

1. Have a class for each supported set of operations. This would probably be a file system type like EXT4 or FAT.
2. Add a method to get a file system's type. This could be done with statfs on Unix-based OSs.

Possible pros:

• Provides a simple way to add easily discoverable APIs in the future.
• The APIs would be tailored to the file system and operation being done instead of trying to support multiple things at once.

Possible cons:

• Need to know the type of the file system you're doing the operation on. This probably won't be too big of a problem because most of the APIs would be file-system-specific anyway.
• Some operations might be common between file systems instead of cleanly fitting in any one file system's class.

[JeremyKuhne]

I think with this one we'd need to do some prototyping to look at how we might design API around the functionality we're looking for. Without concrete code it would be difficult to rationalize a generic API shape. The downside, of course, is that prototyping effort takes time and may not result in shipping code. :)

[Thealexbarney]

Probably adding another flag to EnumerationOptions. At least that is what I would propose as a starting point.

I went ahead and added that to see how it could work, @JeremyKuhne. Thealexbarney@1d090e3

It seems like there would be 2 levels of options here. One would be to add FAT attribute etc. functionality to the Unix build so it matches Windows functionality. I feel like supporting at least FAT features across platforms in the base libraries would be useful because FAT has kind of become the common file system readable by almost everything.

The other would be to have a more generic API shape as discussed above for filesystem-specific operations or maybe platform-specific file operations. This may or may not include things like extended file attributes, info like Unix's stat call, file permissions, etc. That would be a lot more involved and probably be better suited for Platform Extensions.

[Thealexbarney]

This is still something that would be useful to me