GDScript test framework

[miv391]

GDScript would benefit from a built-in test framework. For example, I wrote a function for calculating an intersection of two arrays. It's not a totally trivial function (or at least I have enough programming experience to know that I can miss a few corner cases...), so the function needs to be tested. I can write following kind of code:

func test():
    var arr1 := [1, 6, 3, 1, 7, 3, 4]
    var arr2 := [6, 8, 8, 5, 3]

    assert(intersection([], []) == [])
    assert(intersection(arr1, []) == [])
    assert(intersection([], arr2) == [])
    assert(intersection(arr1, arr2) == [3, 6]

But where and when I call the test function? I don't want to execute it every time I run the game. assert can be used for checking results, but it is not the best tool for the job as the execution stops at first failing assert and the output is not very for useful for inspecting what went wrong.

I propose adding test framework with following features:

• Tests are written in files with extension .test.gd.
• Test cases are written in functions with test_ prefix.
• Test file could have a few special functions like before_all, before_each etc. which can be used to set up the test data. Common stuff in pretty much every test framework.
• Some expect functions to be used instead of assert. Expect failing would not stop execution and it would print out useful information telling why it failed. What the exact syntax of the expect would be is beyond of this proposal.
• When the game is run, *.test.gd files are ignored.
• When the tests are run, only *.test.gd files are executed.
• Test code should be able to access private functions and variables of the testable code. E.g. intersection.test.gd could access private functions and variables in intersection.gd. That way internal functions do not need be exposed just to be testable.
• Test results are printed in a nice format where possible failures are easily seen.

Feature set could be initally kept minimal and new features could be added later. Writing visual tests for GPU shaders should not be an inital goal.

I know there are extensions for testing, but in my experience, the programmers are much more likely to use built-in test features than choosing from competing test extensions requiring additional installs and which may or may not be supported in the future. All extension authors would benefit from built-in test framework as all extensions could contain tests which the user could run if necessary and without installing any other third party extensions.

Testing is not needed only for boring stuff like the array intersection in the example. For example, if I had a procedural dungeon generator and I want that every dungeon has exactly one skeleton king, how do I test that? Playing the game a few times? Or generating a few thousand dungeons and testing the skeleton king count is 1 every time? Any game with any kind of game economy should use testing. If I want that the ratio of monsters to treasures on the dungeon level is always on a certain range, having tests is the only way to make it sure that nothing breaks when making changes to the dungeon generator.

[Xrayez]

See related proposal:

• Make it as easy as possible for game developers to unit test their games #3448

See also related discussion in Goost:

• Built-in unit testing system integrated into the editor goostengine/goost#152

[Xrayez]

I know there are extensions for testing, but in my experience, the programmers are much more likely to use built-in test features than choosing from competing test extensions requiring additional installs and which may or may not be supported in the future.

I kind of have to agree here, especially when I tend to use bleeding edge version of the engine whenever feasible. Most of the time, authors of testing frameworks are not willing to update their code to work with latest version of the engine. In fact, I had to implement my own internal unit testing framework for this.

While the idea is good, I'm not sure about feasibility of it. If it's too simple - nobody will use it. For moderate complexity - someone will use it, however I'm really not sure if Godot core developers would be willing to maintain such a beast.

As you may know, Godot has integrated a third-party unit testing framework for testing C++ codebase:

• [TRACKER] Unit tests to add or improve godot#43440

But as time goes on, I see that using doctest is not really a flexible solution for testing front-end classes, and writing tests becomes much harder. Like this really complex SignalWatcher: https://github.com/godotengine/godot/blob/fe52458154c64fb1b741df4f7bd10106395f7cbd/tests/test_macros.h#L181-L341, which can be implemented a lot easier in GDScript.

Therefore, I do think Godot would benefit from having its own GDScript-based testing framework implemented, at least for covering its own codebase. Then, the same framework could be used by others, and it would be maintained naturally.

I'm currently using GUT for Goost tests, and it works quite well: https://github.com/goostengine/goost/tree/gd3/tests. But the amount of code that I actually need from GUT is likely very minimal, I don't need extra features like doubling, spying, stubbing etc. The core behind GUT is not that complex, and will be useful for most use cases.

By the way, there was a PR that aimed to add a similar system, but it was rejected:

• Adding testing to gdscript module godot#27564