The most basic of any group of tests will be the Unit Test. Unit tests should be to validate the basic logic of the application. For building Unit tests in Erlang the normal tool is EUnit which comes as a standard part of the Erlang distribution.
Tests in EUnit can be created in the same module as the code to be tested or in a separate module. If the tests are in their own module then all the functions to be tested must be exported for the tests to work. However it is possible to compile the module with a export_all directive for testing. This can be done as in this example by using the erlang -ifdef(EUNIT). If the tests are in their own module then that module should be named the same at the module to be tested but with the suffix _test or _tests. So a module square will have tests in square_test.
-ifdef(EUNIT).
-compile(export_all).
-endif.However there is no requirement that there be a one to one correspondence between test modules and production modules. If having the tests for one module spread over two or more files makes sense then EUnit will let you do that.
When building a test module the first thing that you should do is include the file eunit/include/eunit.hrl which contains all the macros that are used by EUnit. After that the test module should define a set of functions each with a name ending in "_test" that define the actual tests.
|
Note
|
Any valid atom is a legal function name in Erlang, so naming a test function something like 'check that function does the right thing _test' is perfectly legal and will work just fine as far as eunit is concerned, though some of the other tools may not work well with it. |
To illustrate this lets define a very simple module square that contains a simple function square/1 that takes a number and squares it.
link:eunit/src/square.erl[role=include]To test this code we need a test script like square_test
link:eunit/src/square_test.erl[role=include]A test will fail if the process exits, this could be from calling the Erlang BIF exit/1 or other error, including a failed pattern match. However using assertions will in general result in a more readable error report so it should be generally preferred.
EUnit features a number of assertions for determining if a test has passed. All of these are implemented as processor macros via the eunit.hrl file.
The most basic assertion is ?assert/1 which will pass if the value is true and fail otherwise. There is also an ?assertNot/1 which will pass if the value is false
The next form is ?assertEqual/2 which takes an expected value and an expression for example ?asssertEqual(4, square:square(2)) will pass and ?assertEqual(5, square:square(2)) will fail.
The converse of the ?assertEqual/2 macro is the ?assertNotEqual/2 macro. This will pass if the two parameters are not equal.
If you want to assert that something matches a pattern you can use the ?assertMatch/2 macro which will assert that the returned value matches a guard expression.
TODO Example
In addition you will want to test for error conditions. There are four macros to test for an error ?assertException/2, assertError/2, assertExit/2, and assertThrow/2.
TODO Example
There are a few ways to run a test suite in EUnit. First of all you can call module:test() to run the tests in the square_test module run the function square_test/0 from the REPL. You can also run tests by calling the function eunit:test/1 and passing the name of the test module. So in this case you can call eunit:test(square_test).
When tests run there are two possible results, the first is that all tests pass in which case you will get a result like this:
All 58 tests passed.If a test fails then EUnit will return a result like this. There are two possible ways a test can fail. The first is that it fails to match an assertion made with one of the EUnit macros, In that case the you will see an error like the first failure here. It will list the line, the expression and other data that will help you figure out what went wrong.
In the second case the test code called exit(test) which causes the test to fail but does not give any of the information that the ?assert() family of macros gives. Any uncaught failure including an Exit will result in a failed test.
my_list_test: in_list_one_element_list_present_test...*failed*
::{assertEqual_failed,[{module,my_list_test},
{line,12},
{expression,"r"},
{expected,3},
{value,r}]}
my_list_test: in_list_two_elements_list_present_test...*failed*
::test
=======================================================
Failed: 2. Skipped: 0. Passed: 57.Normally when running tests all tests that pass are not shown (after all there may be a large number of them). To show them call the function eunit:test/2 with the second parameter being set to [verbose] this will show all the tests and will also show timing information on the tests.
If you have a whole directory of tests and wish to run them as a group you can call the eunit:test/1,2 functions with the name of the directory instead of a file this will cause EUnit to run all tests in a directory.
If you are building your project with rebar then running your tests is as simple as running rebar eunit. It will then build everything and run all the eunit tests in your project. By default if you do this it will run all the tests for any dependencies you may have. Normally you probably do not want to do this as it can take a while to run, so use the command rebar eunit skip_deps=true which will cause rebar to only run the tests of the actual project. You can make the output of rebar tests more verbose with the -v switch, which will do the same as the verbose option above.
Rebar is my preferred method of running tests.
Ideally you want your unit tests to cover as much of your project as you can have. However there is an issue of how do you figure out what parts of your code base are covered? The practical way is to have your code base tell you.If you are running your tests with rebar then you can add the following directive to your rebar.config file and it will create HTML pages that will tell you what is and is not covered.
{cover_enabled, true}.TODO: Example screenshot
If you would like to have coverage in more detail than on a line by line basis than the smother module https://github.com/ramsay-t/smother can provide Modified Decision/Condition Coverage. which is to say that instead of just telling you if a given line of code was executed it can tell you if each conditional on that line had all possible values tried. This can find cases where bugs can creap in despite test coverage due to the fact that a possible condition is masked by other factors
TODO: Example
Often when running tests we want to have some form of setup and clean-up happen before and after each test. For example The setup code could create a Mnesia or ETS table and the clean-up code can delete it. In order to do this we need to use a test generator instead of a basic test.
In fixtures.erl we use have a generator that returns the tuple {setup, setup/0, cleanup/1, ?_test()} which is the actual test. In this case setup/0 and cleanup/1 are called before and after each test. The return value of setup is passed to cleanup, so if you need say delete a table or close a resource you can do it there.
link:eunit/src/fixtures.erl[role=include]If your code is anything like mine you will find that many of your functions depend heavily on external functions, which may come from the Erlang Standard Library or from other modules that may have been written in house or pulled from the web.
In an ideal world a test should be able to isolate your code from the modules it calls so that you can test your code without having to worry about questions like, the status of an external API or the complex behaviour of a library method.
Therefore for any external functions which are not pure in the functional sense we want to replace them with a simple known function that we know will always do what we want so that only the code under test is
|
Note
|
Pure in the functional sense, which is to say that the return value of the function is only a result of the input parameters. You can assume that proplist:get_value/2 will only return values based on the input, while say something in the httpc module will probably require network access, which could cause tests to fail due to things outside your control. |
To do that we use a mocking library such as meck which can take functions in other modules and replace them with test functions that allow a simplified truth table.
Meck provides mocks on the level of the Erlang module, before running a test you must setup mocking for that library, then remove it when done.
For example if you want to mock the webmachine wrq:req_body/1 function you might use code like this in this example.
Here we setup the test with meck:new(wrq) which tells meck to mock that module, then in the body of the test we use meck:expect/3 to tell it when the wrq:req_body/1 function is called to call our test function which returns a set binary to the caller. After this we can call wrq:req_body/1 and know exactly what it will return.
link:meck/src/meck_simple.erl[role=include]The most basic way to include meck in your project is to include it via Rebar, add this to the project’s rebar.config file and run "rebar get_deps" to install it.
{deps, [
{meck,
".*", {git, "git://github.com/eproxus/meck.git", "HEAD"}}]}.The real problem with mocking frameworks like meck is that they violate the idea of DRY. For example if you have two modules, a and b and you change the way a function works in b you must also change the mocks in a_test.
Dialyzer can solve some of this problem. If the contracts between two functions don’t match then dialyzer will find it, even if meck does not.
In addition in the case of meck the order in which rebar compiles your project can matter. If your project consists of many OTP applications in one git repository then if meck can not find a module it will fail. So while in normal compilation order of modules does not matter with meck it does.
Sometimes you will include several sub projects with different functionality, the problem is that each of these will often include meck as well. The problem is that often they will include different versions of meck. The problem with this is that rebar will often not compile when the versions don’t match.
It is possible to create code that requires multiple modules to be mocked in order to test the code. This should be considered a major code smell.
If your functions are too long then, testing them becomes more difficult, in this case it is probably a clue that your code needs to be re-factored. Shorter functions are easier to test and easier to reason about.
If a major part of your code is gluing together external functions that is probably a good sign that the code should be tested not with unit tests but with integration tests and static analysis.
TODO: Expand
There are three places that eunit lets you setup a test. You can place the test in the main file of the module. This has the advantages of the fact that it keeps the tests and the code together, in addition Erlang will run the tests when it loads the module into the runtime, which if you are doing a fast compile/run cycle can be helpful. However it also mixes the test code with the actually code.
A second option is to put the tests in a _test.erl file in the src directory, the problem with this is that the beam files will end up in the ebin directory with the executable code, which is not necessary, I dislike this approach as it tends to create a bunch of executables that may end up getting shipped when they don’t need to be.
My method of choice is to put the tests in the test directory in _test.erl files. When you run the tests with rebar it will put the beam files in the .eunit directory and not in the ebin directory, so there is no chance that these will get executed by mistake when they are not supposed to.
There is always an issue of what to test and what not to test. Some people like to test everything. However some code is hard to test in a way that can prove illuminating. I have found that tests for the initialization function of supervisors is ation function of supervisors is often hard to test in a way that can prove useful for the developer. These things should of course be subject to some form of verification, but it does not have to be a unit test, it can be done in some other type of test.