Proposal: introduce an abstract file system interface into std

[RossComputerGuy]

The problem: people may want to use libraries which could connect to things like Gvfs. The issue is that overriding stuff in root module is necessary to perform this.

The solution:

• Refactor std.fs to include generic things such a path (similar to std.mem but file system stuff)
• Introduce the abstract interface into std.vfs
• Add std.fs.local() to the OS implemented file system interface

This will introduce a lot of incompatibilities but would make it possible to use other file system libraries with libraries which expect Zig's std.fs.

VFS implementations:

• Gvfs: https://wiki.gnome.org/Projects/gvfs
• Rust Vfs crate: https://docs.rs/vfs/latest/vfs/
• Dart's file: https://pub.dev/packages/file

[matu3ba]

to include generic things such a path (similar to std.mem but file system stuff)

1. Can you provide specific use cases for the compiler/package manager?
2. Can you explain the specific use case you want to refactor std.fs for? It would be nice to have some concrete use case you would propose to estimate feasibility + sketch of list of things needed to be changed.

This will introduce a lot of incompatibilities but would make it possible to use other file system libraries with libraries which expect Zig's std.fs.

3. Do you maybe mean breaking changes here or what current supported use cases would not be usable and not be fixable anymore?

Personally my biggest concern would be, if and how file system semantics can and can not be cleanly mapped and how to make these things explicit to the user (say how to make absence of flush needed to write a file an error to make things interchangeably usable).
Virtualization can hide lots of bugs in broken code, which can also be the case for allocation related things in a virtual machine and it would be good to have some best practice (for third parties).

4. What is your position on this or what could/should libstd do with regard to this?

[notcancername]

Can you provide specific use cases for the compiler/package manager?

Unpacking archives using something like the familiar IterableDir interface is a use case for the package manager. Such an interface would also be useful for accessing file systems over a network (FUSE file systems are already doing this: sshfs, curlftpfs).

Virtualization can hide lots of bugs in broken code, which can also be the case for allocation related things in a virtual machine and it would be good to have some best practice (for third parties).

Could you elaborate on this? I don't understand what you mean.

I share your semantics concern, the only solution I can think of is to not guarantee precise semantics of operations.

std.fs.path already appears well-equipped to handle such an interface, disregarding functions that call out to std.os and the whole path separator stuffs. Maybe it should become std.path eventually.

[Khitiara]

an abstract file system interface would also be useful in the context of certain kinds of gamedev resource loading, especially when space-constrained, especially if it could allow nesting/mounting, as that permits even nested archives etc in the asset file tree as needed by the game/application

[matu3ba]

Could you elaborate on this? I don't understand what you mean.

Jamie has an excellent talk on the differences in the context of databases, which has some simple performance examples:

• Kernel in-memory synchronization is much faster than file disk flushes, mentioned that fences in memory are much faster than on disk and follow up slice.

Faster means, that you can get away without or potentially erroneous synchronization in the in-memory file system (msync & others), but not in the hardware-based file system (fsync & others).
The difference is usually only observable, if you access a file system from different processes or close and open a file, because the Kernel with file system also tries to call flush to disk as few as possible.

[Cloudef]

This would be great for tests as well that rely on platform specific search paths and what not, to validate that the search path logic works.

[arkadiuszwojcik]

@matu3ba one use case I would be intrested in would be custom file system for webserver hosted on embedded device. I would like to provide custom filesystem based on LittleFS with files stored in board flash memory.

I think this relates to: #6600

Some inspirations:
RT-Thread DFS
Zephyr FS