Increase default timeout to 60s

[sudara]

Waiting for green build here: https://github.com/sudara/pluginval/actions/runs/2944471209

This PR bumps the default timeout to 60s from 30s.

I'm seeing pluginval regularly killed in CI (sometimes with exit code 137) — it seems to regularly take over 30s to complete at strictness level 10, especially on macOS. 30s seems too optimistic for real-world plugins, especially now with the additional auval stuff added.

Pluginval's own pluginval runs are taking longer than 30s (on somewhat trivial plugins) (not totally sure, but i think they might be getting killed before succeeding too, I'm not seeing that final summary that with "All tests completed," the timing info, etc)

[sudara]

Related: it seems this PR https://github.com/Tracktion/pluginval/pull/81/files removed "closure" that all tests have finished. It used to say "Time taken to run all tests: 52 secs" and now there's nothing to confirm everything went well unless verbose logging is on, which seems unideal.

Not sure what happened the UnitRunner's "All tests completed" though...

[sudara]

Hmm, it seems the timeout gets reset after resultsUpdated and in the longer tests like EditorAutomationTest?

Maybe this change isn't needed (at least until I look more deeply into where the timeouts are occurring...)

[sudara]

This is what I'm see regularly on my macOS strictness level 10 builds, it bails out somewhere in the fuzz testing...

Fuzz testing parameter: 350 - H33 Volume
Fuzz testing parameter: 351 - H33 Delay
/Users/runner/work/_temp/9d2f5c5e-11b7-43ec-be00-08da64eeff31.sh: line 3: 27618 Killed: 9              
pluginval.app/Contents/MacOS/pluginval --strictness-level 10 --timeout-ms 110000 --verbose --validate "***_artefacts/Release/VST3/MyPlugin.vst3"
Error: Process completed with exit code 137.

Even after manually increasing "timeout-ms" it seems to be a problem... not quite sure how to best diagnose...

[drowaudio]

The "Time taken to run all tests: 52 secs" was changed to verbose due to another FR to support snapshot testing. I think that was a mistake. It should probably keep the "All tests completed message" but just add the time as a verbose message just after? You can always log the time of a process on posix using time.

The timeout should reset after any output or logging message. I'm not sure why a fuzz test is taking longer than that?

For the auval and validator tests in non-verbose mode that don't output logs whilst processing, they should probably manually reset the timeout in their run loop where they grab the output from the process.

[sudara]

Ok, not sure how/why or if it's coincidence, but my macOS CI finally completed with the increased default timeout (but not with timeout-ms specified?)

Time taken to run all tests: 58 secs
Completed tests in pluginval / Fuzz parameters

I'll leave this open while I get a few more data points...

[sudara]

Some additional points:

1. GitHub support said "it was killed due to memory usage" but I confirmed mem locally doesn't go above 150MB, very much under the 15GB limit.
2. Exit code 137 just generally means sigkill — this is why GA support would think it was killed due to mem, they monitor mem usage and kill -9 from the outside
3. pluginval is doing something here with kill -9, but I'm not sure what — is it replacing every type of signal with a kill -9? If so, why? https://github.com/Tracktion/pluginval/blob/develop/Source/CommandLine.cpp#L45

Basically, I'm almost certain pluginval is killing itself, but given there's no logging and it's a kill -9, it's almost impossible to discern the "why"

[sudara]

Closing this. It wasn't a timeout, but a crash swallowed by custom crash handling code on macOS, clarified in #98