Ethereum EIP-4844: Go wrapper for KZG commitments

[mratsim]

This is a follow-up of #304.

It adds a go wrapper for Ethereum KZG commitments.

In particular this will allow differential fuzzing vs c-kzg-4844 and go-kzg-4844 in kzg-fuzz cc @jtraglia.

Some peculiarities:

• On Windows, Go cannot statically link vs .lib file directly like C, Nim or Rust. We need to use -Wl,-Bstatic -lconstantine -Wl,-Bdynamic as a workaround. Fix raised upstream cmd/go/internal/work/security.go: allow direct linking of Windows .lib static libraries golang/go#64679.
  Note that that workaround doesn't work on MacOS due to Xcode/Apple Clang not supporting the flag.
• On MacOS, it seems like this fscanf requires an extra byte in the character buffer or it triggers AddressSanitizer or a stack check fail from the fstack-protector:

  constantine/constantine/trusted_setups/ethereum_kzg_srs.nim

  Lines 201 to 206 in 1f3e6b1

  	# fscanf for up to 4 digits. fscanf ignores whitespaces and \r when parsing an int
  	const parseInt32 = "%4u\n"
  	# fscanf for up to 96 chars, up until EOL, reporting number read, with EOL skipping
  	const readHexG1 = "%96[^\r\n]%n%*[\r\n]"
  	# fscanf for up to 192 chars, up until EOL, reporting number read, with EOL skipping
  	const readHexG2 = "%192[^\r\n]%n%*[\r\n]"

Running go, using the optional CC=clang for maximum performance:

git clone https://github.com/mratsim/constantine
cd constantine
CC=clang nimble make_lib
cd constantine-go
go test -modfile=../go_test.mod

Constantine comes with a separate go_test.mod for extra test dependencies. Test dependencies must be specified with -modfile=../go_test.mod. The library itself is dependency-free.

For the trusted setup, Constantine supports the same format as c-kzg-4844.

[jtraglia]

LGTM 👍 just two small nits.

[jtraglia]

This is a good find by the sanitizer. It's right. Because you're reading a string (rather than a byte like it's done in c-kzg-4844) it will add a null terminator at the end.

[mratsim]

I actually open files in binary mode not text (wb):

constantine/constantine/platforms/fileio.nim

Lines 40 to 57 in c73ce0e

	const
	childProcNoInherit = block:
	# Child processes should not inherit open files
	when defined(windows):
	"N"
	elif defined(linux) or defined(bsd):
	"e"
	else:
	""
	MapFileMode = [
	# We want to ensure no dynamic alloc at runtime with strings
	kRead: cstring("rb" & childProcNoInherit),
	kOverwrite: cstring("wb" & childProcNoInherit),
	kAppend: cstring("ab" & childProcNoInherit),
	kReadWrite: cstring("rb+" & childProcNoInherit),
	kReadOverwrite: cstring("wb+" & childProcNoInherit)
	]

And the modifier for strings is %s, which doesn't appear in const readHexG1 = "%96[^\r\n]%n%*[\r\n]"
In fact, the sanitizer on Linux doesn't complain.

[jtraglia]

It shouldn't matter that it's opened in binary mode. It just matters that it's reading a string.

Here's an example with sscanf which should operate the same:

#include <stdio.h>

int main(void)
{
    char buffer[97];
    int i, n;
    const char *staticString = \
        "0123456789" // 10
        "0123456789" // 20
        "0123456789" // 30
        "0123456789" // 40
        "0123456789" // 50
        "0123456789" // 60
        "0123456789" // 70
        "0123456789" // 80
        "0123456789" // 90
        "012345";    // 96

    for (i = 0; i < 97; i++) {
        buffer[i] = 0xFF;
    }

    printf("\nInitial buffer contents:");
    for (i = 0; i < 97; i++) {
        if (i % 10 == 0) printf("\n");
        printf("%02X ", (unsigned char)buffer[i]);
    }
    printf("\n");

    sscanf(staticString, "%96[^\r\n]%n%*[\r\n]", buffer, &n);

    printf("\nBuffer contents after sscanf:");
    for (i = 0; i < 97; i++) {
        if (i % 10 == 0) printf("\n");
        printf("%02X ", (unsigned char)buffer[i]);
    }
    printf("\n");

    printf("\nNumber of characters read: %d\n", n);

    return 0;
}

For me this is the output:

$ make test && ./test
cc  -I/opt/homebrew/include/ -L/opt/homebrew/include/ -L/opt/homebrew/lib/  test.c   -o test

Initial buffer contents:
FF FF FF FF FF FF FF FF FF FF
FF FF FF FF FF FF FF FF FF FF
FF FF FF FF FF FF FF FF FF FF
FF FF FF FF FF FF FF FF FF FF
FF FF FF FF FF FF FF FF FF FF
FF FF FF FF FF FF FF FF FF FF
FF FF FF FF FF FF FF FF FF FF
FF FF FF FF FF FF FF FF FF FF
FF FF FF FF FF FF FF FF FF FF
FF FF FF FF FF FF FF

Buffer contents after sscanf:
30 31 32 33 34 35 36 37 38 39
30 31 32 33 34 35 36 37 38 39
30 31 32 33 34 35 36 37 38 39
30 31 32 33 34 35 36 37 38 39
30 31 32 33 34 35 36 37 38 39
30 31 32 33 34 35 36 37 38 39
30 31 32 33 34 35 36 37 38 39
30 31 32 33 34 35 36 37 38 39
30 31 32 33 34 35 36 37 38 39
30 31 32 33 34 35 00

Number of characters read: 96

As you can see, the 97th byte has been set to 0 but only 96 bytes were read.

[jtraglia]

And the modifier for strings is %s, which doesn't appear in const readHexG1 = "%96[^\r\n]%n%*[\r\n]"
In fact, the sanitizer on Linux doesn't complain.

Oops didn't respond to this. It doesn't need to be %s for it to be recognized as a string.