Skip to content

Getting Started With Fuzzing

Jump to bottom
Brian Stafford edited this page Jul 27, 2024 · 3 revisions

Getting Started With Fuzzing

Go started supporting fuzzing in its standard toolchain beginning in Go 1.18. Fuzzing is a type of automated testing which continuously manipulates inputs to a program to find issues such as panics or bugs. These semi-random data mutations can discover new code coverage that existing unit tests may miss, and uncover edge case bugs which would otherwise go unnoticed. Since fuzzing can reach these edge cases, fuzz testing is particularly valuable for finding security exploits and vulnerabilities.

Writing a Fuzz Test

A fuzz test must be in a *_test.go file as a function in the form FuzzXxx. This function must be passed a*testing.F argument, much like a *testing.T argument is passed to a TestXxx function.

Below is an example of a fuzz test that’s testing the behaviour of dex/encode/encrypt.

// Fuzz name must be `FuzzXXX` which accepts only a `*testing.F`, and has no return value.
func FuzzDecrypt(f *testing.F) {

	// Providing seed corpus helps the fuzzing engine to generate new seeds efficiently.
	seeds := []struct {
		b []byte
		n int
	}{{
		n: 200,
		b: []byte("4kliaOCha2longerbyte"),
	}, {
		n: 20,
		b: []byte("short123456"),
	}, {
		n: 50,
		b: []byte("23Fgfge34"),
	}, {
		n: 10,
		b: []byte("asdf$#@*(gth#4"),
	}}

	for _, seed := range seeds {
		f.Add(seed.n, seed.b) // Use f.Add to add the seed corpus in the same order as the fuzz target arguments.
	}

	// A fuzz target must be a function without a return value, which accepts a
	// `*testing.T` as the first parameter, followed by the fuzzing arguments.
	// There must be exactly one fuzz target per fuzz test.

	// `f.Fuzz` accepts a fuzz target and runs fuzzing with randomly generated values
	// based on the seed corpus provided. All seed corpus entries must have types
	// which are identical to the fuzzing arguments, in the same order. This is true
	// for calls to `(*testing.F).Add`.
	f.Fuzz(func(t *testing.T, n int, b []byte) {
		// Inputs can be validated and skipped if they are not suitable.
		// This test will panic if len(b) or n is greater than encode.MaxDataLen.
		if n < 1 || n > encode.MaxDataLen || len(b) > encode.MaxDataLen {
			t.Skip()
		}
		crypter := NewCrypter(b)
		thing := randB(n)
		encThing, err := crypter.Encrypt(thing)
		if err != nil {
			t.Fatalf("Encrypt error: %v", err)
		}
		reThing, err := crypter.Decrypt(encThing)
		if err != nil {
			t.Fatalf("Decrypt error: %v", err)
		}
		if !bytes.Equal(thing, reThing) {
			t.Fatalf("%x != %x", thing, reThing)
		}
	})
}

The fuzzing arguments can only be the following types:

  • string, []byte
  • int, int8, int16, int32/rune, int64
  • uint, uint8/byte, uint16, uint32, uint64
  • float32, float64
  • bool

Running Fuzz Tests

There are two modes of running fuzz tests:

  • As a unit test (default go test). Fuzz tests are run much like a unit test by default. Each seed corpus entry will be tested against the fuzz target, reporting any failures before exiting. Failed Fuzz inputs generated by the fuzzing engine for that particular fuzz target are re-run against the fuzz target as part of the default go test.

  • With fuzzing go test -fuzz=FuzzTestName. Each seed corpus entry will be tested against the fuzz target. Also, If there are seed corpus already generated for the fuzz target in $GOCACHE they will be run against the fuzz target, and then the fuzzing engine will continue to generate and test seed corpus until -fuzztime. See Fuzz Flags.

To enable fuzzing, run go test with the -fuzz flag, providing a test name (e.g FuzzTestName) or regex matching a single fuzz test. By default, all other tests in that package will run before fuzzing begins. This is to ensure that fuzzing won’t report any issues that would already be caught by an existing test.

Note: Fuzzing cannot be run for multiple packages at the same time using the -fuzz flag. You must specify at most one Fuzz test (e.g -fuzz=FuzzXXX) [See: support for multiple Fuzz tests].

Command Line Output

  • elapsed: the amount of time that has elapsed since the process began
  • execs: the total number of inputs that have been run against the fuzz target (with an average execs/sec since the last log line)
  • new interesting: the total number of “interesting” inputs that have been added to the generated corpus during this fuzzing execution (with the total size of the entire corpus)
$ go test -v ./... --fuzz=FuzzDecrypt --fuzztime=10x 
=== RUN   TestDecrypt
--- PASS: TestDecrypt (0.04s)
=== RUN   TestSerialize
--- PASS: TestSerialize (0.31s)
=== RUN   TestRandomness
--- PASS: TestRandomness (0.08s)
=== FUZZ  FuzzDecrypt
fuzz: elapsed: 0s, gathering baseline coverage: 0/7 completed
fuzz: elapsed: 0s, gathering baseline coverage: 7/7 completed, now fuzzing with 8 workers
fuzz: elapsed: 1s, execs: 10 (18/sec), new interesting: 2 (total: 9)
--- PASS: FuzzDecrypt (0.57s)
PASS
ok      decred.org/dcrdex/dex/encrypt   1.172s

Failing Input

Failing inputs generated by the fuzzing engine are saved to the main directory of the project (e.g dcrdex/testData/fuzz/$fuzzTarget), and are re-run as part of go test (without -fuzz FuzzTarget) for $fuzzTarget.

A failure may occur while fuzzing for several reasons:

  • A panic occurred in the code or the test.
  • The fuzz target called t.Fail, either directly or through methods such as t.Error or t.Fatal.
  • A non-recoverable error occurred, such as an os.Exit or stack overflow.
  • The fuzz target took too long to complete. Currently, the timeout for an execution of a fuzz target is 1 second. This may fail due to a deadlock or infinite loop, or from intended behaviour in the code.

Other Highlighted Fuzz Flags

  • -fuzztime: The total time or number of iterations that the fuzz target will be executed before exiting, specified as a time.Duration (for example, -fuzztime 1h30s) or using a special syntax Nx to run the fuzz target N times (for example, -fuzztime 1000x). The default is to run forever.
  • -fuzzminimizetime: the time or number of iterations that the fuzz target will be executed during each minimization attempt, default 60sec. You can completely disable minimization by setting -fuzzminimizetime 0 when fuzzing.
  • -parallel: the number of fuzzing processes running at once, default $GOMAXPROCS. Currently, setting -cpu during fuzzing has no effect.
  • -keepfuzzing: Keep running the fuzz test if a crasher is found. (default false)

Best Practices

  • Fuzz targets should be fast and deterministic so the fuzzing engine can work efficiently, and new failures and code coverage can be easily reproduced. Fuzzing is most effective with the most primitive methods and functions. As such, this will take some careful targeting and potentially some very light refactoring here and there to get fuzzable functions.

  • Since the fuzz target is invoked in parallel across multiple workers and in nondeterministic order, the state of a fuzz target should not persist past the end of each call, and the behaviour of a fuzz target should not depend on a global state.

  • Keeping auto-generated seed corpus in $GOCACHE increases the efficiency of the fuzzing engine. The more seed corpus the better.

  • Provide sufficient seed corpus. This could be in the test file, another file or a directory. This will be a guide to the fuzzing engine in generating sensible random inputs. To convert seed corpus from another file or directory to Go fuzzing corpus format use file2fuzz:

      $ go install golang.org/x/tools/cmd/file2fuzz@latest
  $ file2fuzz

Clean-up

  • Fuzz tests can do clean-up using f.Clean(f func()).
  • Use go clean -fuzzcache to remove files stored in the Go build cache for fuzz testing aka seed corpus. The fuzzing engine caches seed corpus files that expand code coverage, so removing these seed corpus may make fuzzing less effective until new inputs are found that provide the same coverage. These files are distinct from those stored in testData directory i.e dcrdex/testData/fuzz dir; clean does not remove those files.

References

Clone this wiki locally