Exposes sys.Stat_t as a portable alternative to syscall.Stat_t (tetra…

…telabs#1567)

Signed-off-by: Adrian Cole <[email protected]>

RATIONALE.md

@@ -1171,6 +1171,68 @@ See https://github.com/WebAssembly/stack-switching/discussions/38
 See https://github.com/WebAssembly/wasi-threads#what-can-be-skipped
 See https://slinkydeveloper.com/Kubernetes-controllers-A-New-Hope/
 
+## sys.Stat_t
+
+We expose `stat` information as `sys.Stat_t`, like `syscall.Stat_t` except
+defined without build constraints. For example, you can use `sys.Stat_t` on
+`GOOS=windows` which doesn't define `syscall.Stat_t`.
+
+The first use case of this is to return inodes from `fs.FileInfo` without
+relying on platform-specifics. For example, a user could return `*sys.Stat_t`
+from `info.Sys()` and define a non-zero inode for a virtual file, or map a
+real inode to a virtual one.
+
+Notable choices per field are listed below, where `sys.Stat_t` is unlike
+`syscall.Stat_t` on `GOOS=linux`, or needs clarification. One common issue
+not repeated below is that numeric fields are 64-bit when at least one platform
+defines it that large. Also, zero values are equivalent to nil or absent.
+
+* `Dev` and `Ino` (`Inode`) are both defined unsigned as they are defined
+  opaque, and most `syscall.Stat_t` also defined them unsigned. There are
+  separate sections in this document discussing the impact of zero in `Ino`.
+* `Mode` is defined as a `fs.FileMode` even though that is not defined in POSIX
+  and will not map to all possible values. This is because the current use is
+  WASI, which doesn't define any types or features not already supported. By
+  using `fs.FileMode`, we can re-use routine experience in Go.
+* `NLink` is unsigned because it is defined that way in `syscall.Stat_t`: there
+  can never be less than zero links to a file. We suggest defaulting to 1 in
+  conversions when information is not knowable because at least that many links
+  exist.
+* `Size` is signed because it is defined that way in `syscall.Stat_t`: while
+  regular files and directories will always be non-negative, irregular files
+  are possibly negative or not defined. Notably sparse files are known to
+  return negative values.
+* `Atim`, `Mtim` and `Ctim` are signed because they are defined that way in
+  `syscall.Stat_t`: Negative values are time before 1970. The resolution is
+  nanosecond because that's the maximum resolution currently supported in Go.
+
+### Why do we use `sys.EpochNanos` instead of `time.Time` or similar?
+
+To simplify documentation, we defined a type alias `sys.EpochNanos` for int64.
+`time.Time` is a data structure, and we could have used this for
+`syscall.Stat_t` time values. The most important reason we do not is conversion
+penalty deriving time from common types.
+
+The most common ABI used in `wasip2`. This, and compatible ABI such as `wasix`,
+encode timestamps in memory as a 64-bit number. If we used `time.Time`, we
+would have to convert an underlying type like `syscall.Timespec` to `time.Time`
+only to later have to call `.UnixNano()` to convert it back to a 64-bit number.
+
+In the future, the component model module "wasi-filesystem" may represent stat
+timestamps with a type shared with "wasi-clocks", abstractly structured similar
+to `time.Time`. However, component model intentionally does not define an ABI.
+It is likely that the canonical ABI for timestamp will be in two parts, but it
+is not required for it to be intermediately represented this way. A utility
+like `syscall.NsecToTimespec` could split an int64 so that it could be written
+to memory as 96 bytes (int64, int32), without allocating a struct.
+
+Finally, some may confuse epoch nanoseconds with 32-bit epoch seconds. While
+32-bit epoch seconds has "The year 2038" problem, epoch nanoseconds has
+"The Year 2262" problem, which is even less concerning for this library. If
+the Go programming language and wazero exist in the 2200's, we can make a major
+version increment to adjust the `sys.EpochNanos` approach. Meanwhile, we have
+faster code.
+
 ## poll_oneoff
 
 `poll_oneoff` is a WASI API for waiting for I/O events on multiple handles.

fsconfig.go

@@ -118,6 +118,18 @@ type FSConfig interface {
 	// advise using WithDirMount instead. There will be behavior differences
 	// between os.DirFS and WithDirMount, as the latter biases towards what's
 	// expected from WASI implementations.
+	//
+	// # Custom fs.FileInfo
+	//
+	// The underlying implementation supports data not usually in fs.FileInfo
+	// when `info.Sys` returns *sys.Stat_t. For example, a custom fs.FS can use
+	// this approach to generate or mask sys.Inode data. Such a filesystem
+	// needs to decorate any functions that can return fs.FileInfo:
+	//
+	//   - `Stat` as defined on `fs.File` (always)
+	//   - `Readdir` as defined on `os.File` (if defined)
+	//
+	// See sys.NewStat_t for examples.
 	WithFSMount(fs fs.FS, guestPath string) FSConfig
 }
 

fsconfig_example_test.go

@@ -14,7 +14,7 @@ var testdataIndex embed.FS
 var moduleConfig wazero.ModuleConfig
 
 // This example shows how to configure an embed.FS.
-func Example_withFSConfig_embedFS() {
+func Example_fsConfig() {
 	// Strip the embedded path testdata/
 	rooted, err := fs.Sub(testdataIndex, "testdata")
 	if err != nil {

imports/wasi_snapshot_preview1/fs.go

@@ -17,6 +17,7 @@ import (
 	"github.com/tetratelabs/wazero/internal/sysfs"
 	"github.com/tetratelabs/wazero/internal/wasip1"
 	"github.com/tetratelabs/wazero/internal/wasm"
+	sysapi "github.com/tetratelabs/wazero/sys"
 )
 
 // fdAdvise is the WASI function named FdAdviseName which provides file
@@ -196,7 +197,7 @@ func fdFdstatGetFn(_ context.Context, mod api.Module, params []uint64) syscall.E
 	}
 
 	var fdflags uint16
-	var st fsapi.Stat_t
+	var st sysapi.Stat_t
 	var errno syscall.Errno
 	f, ok := fsc.LookupFile(fd)
 	if !ok {
@@ -445,7 +446,7 @@ func getWasiFiletype(fm fs.FileMode) uint8 {
 	}
 }
 
-func writeFilestat(buf []byte, st *fsapi.Stat_t, ftype uint8) (errno syscall.Errno) {
+func writeFilestat(buf []byte, st *sysapi.Stat_t, ftype uint8) (errno syscall.Errno) {
 	le.PutUint64(buf, st.Dev)
 	le.PutUint64(buf[8:], st.Ino)
 	le.PutUint64(buf[16:], uint64(ftype))
@@ -1018,7 +1019,7 @@ func writeDirents(buf []byte, dirents []fsapi.Dirent, d_next uint64, direntCount
 }
 
 // writeDirent writes DirentSize bytes
-func writeDirent(buf []byte, dNext uint64, ino fsapi.Ino, dNamlen uint32, dType fs.FileMode) {
+func writeDirent(buf []byte, dNext uint64, ino sysapi.Inode, dNamlen uint32, dType fs.FileMode) {
 	le.PutUint64(buf, dNext)        // d_next
 	le.PutUint64(buf[8:], ino)      // d_ino
 	le.PutUint32(buf[16:], dNamlen) // d_namlen
@@ -1399,7 +1400,7 @@ func pathFilestatGetFn(_ context.Context, mod api.Module, params []uint64) sysca
 	}
 
 	// Stat the file without allocating a file descriptor.
-	var st fsapi.Stat_t
+	var st sysapi.Stat_t
 
 	if (flags & wasip1.LOOKUP_SYMLINK_FOLLOW) == 0 {
 		st, errno = preopen.Lstat(pathName)

imports/wasi_snapshot_preview1/fs_test.go

@@ -27,6 +27,7 @@ import (
 	"github.com/tetratelabs/wazero/internal/u64"
 	"github.com/tetratelabs/wazero/internal/wasip1"
 	"github.com/tetratelabs/wazero/internal/wasm"
+	sysapi "github.com/tetratelabs/wazero/sys"
 )
 
 func Test_fdAdvise(t *testing.T) {
@@ -3518,7 +3519,7 @@ func Test_pathFilestatSetTimes(t *testing.T) {
 			sys := mod.(*wasm.ModuleInstance).Sys
 			fsc := sys.FS()
 
-			var oldSt fsapi.Stat_t
+			var oldSt sysapi.Stat_t
 			var errno syscall.Errno
 			if tc.expectedErrno == wasip1.ErrnoSuccess {
 				oldSt, errno = fsc.RootFS().Stat(pathName)

imports/wasi_snapshot_preview1/poll_test.go

@@ -14,6 +14,7 @@ import (
 	"github.com/tetratelabs/wazero/internal/testing/require"
 	"github.com/tetratelabs/wazero/internal/wasip1"
 	"github.com/tetratelabs/wazero/internal/wasm"
+	sysapi "github.com/tetratelabs/wazero/sys"
 )
 
 func Test_pollOneoff(t *testing.T) {
@@ -536,8 +537,8 @@ var fdReadSub = fdReadSubFd(byte(sys.FdStdin))
 type ttyStat struct{}
 
 // Stat implements the same method as documented on fsapi.File
-func (ttyStat) Stat() (fsapi.Stat_t, syscall.Errno) {
-	return fsapi.Stat_t{
+func (ttyStat) Stat() (sysapi.Stat_t, syscall.Errno) {
+	return sysapi.Stat_t{
 		Mode:  fs.ModeDevice | fs.ModeCharDevice,
 		Nlink: 1,
 	}, 0

internal/fsapi/dir.go

@@ -5,18 +5,9 @@ import (
 	"io/fs"
 	"syscall"
 	"time"
-)
 
-// Ino is the file serial number, or zero if unknown.
-//
-// The inode is used for a file equivalence, like os.SameFile, so any constant
-// value will interfere that.
-//
-// When zero is returned by File.Readdir, certain callers will fan-out to
-// File.Stat to retrieve a non-zero value. Callers using this for darwin's
-// definition of `getdirentries` conflate zero `d_fileno` with a deleted file
-// and skip the entry. See /RATIONALE.md for more on this.
-type Ino = uint64
+	"github.com/tetratelabs/wazero/sys"
+)
 
 // FileType is fs.FileMode masked on fs.ModeType. For example, zero is a
 // regular file, fs.ModeDir is a directory and fs.ModeIrregular is unknown.
@@ -38,7 +29,7 @@ type FileType = fs.FileMode
 type Dirent struct {
 	// Ino is the file serial number, or zero if not available. See Ino for
 	// more details including impact returning a zero value.
-	Ino Ino
+	Ino sys.Inode
 
 	// Name is the base name of the directory entry. Empty is invalid.
 	Name string

internal/fsapi/file.go

@@ -3,6 +3,8 @@ package fsapi
 import (
 	"syscall"
 	"time"
+
+	"github.com/tetratelabs/wazero/sys"
 )
 
 // File is a writeable fs.File bridge backed by syscall functions needed for ABI
@@ -55,7 +57,7 @@ type File interface {
 	//
 	//   - Implementations should cache this result.
 	//   - This combined with Dev can implement os.SameFile.
-	Ino() (Ino, syscall.Errno)
+	Ino() (sys.Inode, syscall.Errno)
 
 	// IsDir returns true if this file is a directory or an error there was an
 	// error retrieving this information.
@@ -132,7 +134,7 @@ type File interface {
 	//   - A fs.FileInfo backed implementation sets atim, mtim and ctim to the
 	//     same value.
 	//   - Windows allows you to stat a closed directory.
-	Stat() (Stat_t, syscall.Errno)
+	Stat() (sys.Stat_t, syscall.Errno)
 
 	// Read attempts to read all bytes in the file into `buf`, and returns the
 	// count read even on error.

internal/fsapi/fs.go

@@ -3,6 +3,8 @@ package fsapi
 import (
 	"io/fs"
 	"syscall"
+
+	"github.com/tetratelabs/wazero/sys"
 )
 
 // FS is a writeable fs.FS bridge backed by syscall functions needed for ABI
@@ -79,7 +81,7 @@ type FS interface {
 	//     same value.
 	//   - When the path is a symbolic link, the stat returned is for the link,
 	//     not the file it refers to.
-	Lstat(path string) (Stat_t, syscall.Errno)
+	Lstat(path string) (sys.Stat_t, syscall.Errno)
 
 	// Stat gets file status.
 	//
@@ -99,7 +101,7 @@ type FS interface {
 	//     same value.
 	//   - When the path is a symbolic link, the stat returned is for the file
 	//     it refers to.
-	Stat(path string) (Stat_t, syscall.Errno)
+	Stat(path string) (sys.Stat_t, syscall.Errno)
 
 	// Mkdir makes a directory.
 	//

internal/fsapi/unimplemented.go

@@ -4,6 +4,8 @@ import (
 	"io/fs"
 	"syscall"
 	"time"
+
+	"github.com/tetratelabs/wazero/sys"
 )
 
 // UnimplementedFS is an FS that returns syscall.ENOSYS for all functions,
@@ -26,13 +28,13 @@ func (UnimplementedFS) OpenFile(path string, flag int, perm fs.FileMode) (File,
 }
 
 // Lstat implements FS.Lstat
-func (UnimplementedFS) Lstat(path string) (Stat_t, syscall.Errno) {
-	return Stat_t{}, syscall.ENOSYS
+func (UnimplementedFS) Lstat(path string) (sys.Stat_t, syscall.Errno) {
+	return sys.Stat_t{}, syscall.ENOSYS
 }
 
 // Stat implements FS.Stat
-func (UnimplementedFS) Stat(path string) (Stat_t, syscall.Errno) {
-	return Stat_t{}, syscall.ENOSYS
+func (UnimplementedFS) Stat(path string) (sys.Stat_t, syscall.Errno) {
+	return sys.Stat_t{}, syscall.ENOSYS
 }
 
 // Readlink implements FS.Readlink
@@ -97,7 +99,7 @@ func (UnimplementedFile) Dev() (uint64, syscall.Errno) {
 }
 
 // Ino implements File.Ino
-func (UnimplementedFile) Ino() (Ino, syscall.Errno) {
+func (UnimplementedFile) Ino() (sys.Inode, syscall.Errno) {
 	return 0, 0
 }
 
@@ -127,8 +129,8 @@ func (UnimplementedFile) SetNonblock(bool) syscall.Errno {
 }
 
 // Stat implements File.Stat
-func (UnimplementedFile) Stat() (Stat_t, syscall.Errno) {
-	return Stat_t{}, syscall.ENOSYS
+func (UnimplementedFile) Stat() (sys.Stat_t, syscall.Errno) {
+	return sys.Stat_t{}, syscall.ENOSYS
 }
 
 // Read implements File.Read

internal/fstest/times_notwindows.go

@@ -4,6 +4,6 @@ package fstest
 
 import "io/fs"
 
-func timesFromFileInfo(t fs.FileInfo) (atim, mtime int64) {
+func timesFromFileInfo(fs.FileInfo) (atim, mtime int64) {
 	panic("unexpected")
 }

internal/fstest/times_windows.go

@@ -5,8 +5,8 @@ import (
 	"syscall"
 )
 
-func timesFromFileInfo(t fs.FileInfo) (atim, mtime int64) {
-	if d, ok := t.Sys().(*syscall.Win32FileAttributeData); ok {
+func timesFromFileInfo(info fs.FileInfo) (atim, mtime int64) {
+	if d, ok := info.Sys().(*syscall.Win32FileAttributeData); ok {
 		return d.LastAccessTime.Nanoseconds(), d.LastWriteTime.Nanoseconds()
 	} else {
 		panic("unexpected")

internal/gojs/fs.go

@@ -7,12 +7,12 @@ import (
 	"syscall"
 
 	"github.com/tetratelabs/wazero/api"
-	"github.com/tetratelabs/wazero/internal/fsapi"
 	"github.com/tetratelabs/wazero/internal/gojs/custom"
 	"github.com/tetratelabs/wazero/internal/gojs/goos"
 	"github.com/tetratelabs/wazero/internal/gojs/util"
 	internalsys "github.com/tetratelabs/wazero/internal/sys"
 	"github.com/tetratelabs/wazero/internal/wasm"
+	"github.com/tetratelabs/wazero/sys"
 )
 
 var (
@@ -184,7 +184,7 @@ func syscallFstat(fsc *internalsys.FSContext, fd int32) (*jsSt, error) {
 	}
 }
 
-func newJsSt(st fsapi.Stat_t) *jsSt {
+func newJsSt(st sys.Stat_t) *jsSt {
 	ret := &jsSt{}
 	ret.isDir = st.Mode.IsDir()
 	ret.dev = st.Dev