Better error handling - panic instead of "failing up"

[rnbwdsh]

If the gotestsum isn't installed or your $GOPATH isn't in your $PATH, you'll fail to call gotestsum after querying your model.

Here is an example output. The part that's actually relevant is marked as bold and it's kind of in the middle of the other "spam" that's not really relevant.

Another standard error close to this position was, that my generated program tried to include libs that I don't have installed.

GOROOT=/usr/lib/go #gosetup
GOPATH=/usr/lib/go/bin #gosetup
/usr/lib/go/bin/go build -o /home/m/.cache/JetBrains/IntelliJIdea2023.2/tmp/GoLand/___2codegemma_7b github.com/symflower/eval-dev-quality/cmd/eval-dev-quality #gosetup
/home/m/.cache/JetBrains/IntelliJIdea2023.2/tmp/GoLand/___2codegemma_7b evaluate --model ollama/codegemma:7b
2024/04/11 13:02:26 Checking that models and languages can be used for evaluation
2024/04/11 13:02:26 Evaluating model "ollama/codegemma:7b" using language "golang" and repository "/home/m/IdeaProjects/eval-dev-quality/testdata/golang/plain"
2024/04/11 13:02:30 Model "ollama/codegemma:7b" responded to query Given the following Go code file "plain.go" with package "plain", provide a test file with the same structure and package="plain" for this code.
The tests should produce 100 percent code coverage and must compile.
The response must contain only the raw test code and nothing else.

    ```golang
    package plain

    func plain() {
            return // This does not do anything but it gives us a line to cover.
    }
    ```
       
       Start your answer with tripple backticks.

with: ```go
package plain

    import "testing"

    func TestPlain(t *testing.T) {
            plain()
    }
    ```

2024/04/11 13:02:30 $ gotestsum --format standard-verbose --hide-summary skipped -- -cover -v -vet=off ./...
2024/04/11 13:02:30 Evaluated model "ollama/codegemma:7b" using language "golang" and repository "/home/m/IdeaProjects/eval-dev-quality/testdata/golang/plain": encountered 1 problems
2024/04/11 13:02:30 Excluding model "ollama/codegemma:7b" since it was not able to solve the "plain" repository for language "golang": [exec: "gotestsum": executable file not found in $PATH
github.com/symflower/eval-dev-quality/language.(*LanguageGolang).Execute
/home/m/IdeaProjects/eval-dev-quality/language/golang.go:82
github.com/symflower/eval-dev-quality/evaluate.EvaluateRepository
/home/m/IdeaProjects/eval-dev-quality/evaluate/repository.go:55
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.(*Evaluate).Execute
/home/m/IdeaProjects/eval-dev-quality/cmd/eval-dev-quality/cmd/evaluate.go:117
github.com/jessevdk/go-flags.(*Parser).ParseArgs
/usr/lib/go/bin/pkg/mod/github.com/jessevdk/[email protected]/parser.go:335
github.com/jessevdk/go-flags.(*Parser).Parse
/usr/lib/go/bin/pkg/mod/github.com/jessevdk/[email protected]/parser.go:191
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.Execute
/home/m/IdeaProjects/eval-dev-quality/cmd/eval-dev-quality/cmd/command.go:26
main.main
/home/m/IdeaProjects/eval-dev-quality/cmd/eval-dev-quality/main.go:8
runtime.main
/usr/lib/go/src/runtime/proc.go:271
runtime.goexit
/usr/lib/go/src/runtime/asm_amd64.s:1695
plain.go]
2024/04/11 13:02:30 Evaluating models and languages
2024/04/11 13:02:30 Evaluation score for "ollama/codegemma:7b": #executed=0.0%(0/1), #problems=100.0%(1/1), average statement coverage=0.0%

Calling with a symflower model when you don't have the symflower stuff installed also doesn't fail very gracefully.

[zimmski]

There are two parts to this issue:

• Doing panics instead of using os.Exit for when we want to fail hard. Because the later does not allow for good testing. Java language implementation #62
• Dong a "tool check" for every language to make sure that tools are installed and working before doing the evaluation.

[zimmski]

This has been implemented and with the container image all problems should be gone anyway.