Refactor internal architecture of Alsatian

[pathurs]

I've created a Proof of Concept (POC) here:

https://github.com/pathurs/alsatian-refactor-POC

This would be an internal refactor of Alsatian's main processing stream, and would hopefully be able to solve many of the issues.

Improvements over current Alsatian architecture:

• Implicitly handle promises Expect().toThrowErrorAsync() not executing function and passing silently. #458 Handle all asynchronous errors #368 Expect.toThrow for async functions fails when an error is thrown #353
• Continue a TestCase when an expectation fails Display the Line number of failed Expect #366 Multiple failures on a single Test Run #225
• Runs Tests Asynchronously Ability to run tests in parallel #49
• Individually defined matchers Rework Expect #473
• Describe an expectation Custom error messages from Expect failures #227
• Modularisation Split out spying into separate library #192 Investigate splitting up into modules #302 Rework Expect #473
• More...?

and related to #455 #436

My code was made in only a few hours, so it does not match the same API with decorators, however I hope you can imagine it would work the same way, but with improved handling.

@jamesadarich @Jameskmonger fyi

[pathurs]

The most notable improvement is used in the example.

Notice in the below extract, that the 2 Testcases and their individual expectations are run asynchronously.

Code

runner.addTest(
    new TestCase(() => {
        Expect(
            new Promise((resolve, reject) => {
                setTimeout(() => {
                    reject();
                }, 100);
            })
        ).toResolve('reject after 100 milliseconds should resolve'); // Fail
        Expect(
            new Promise((resolve, reject) => {
                setTimeout(() => {
                    resolve();
                }, 20);
            })
        ).toResolve('resolve after 20 milliseconds should resolve'); // Pass
    })
);

runner.addTest(
    new TestCase(() => {
        Expect(
            new Promise((resolve, reject) => {
                setTimeout(() => {
                    reject();
                }, 10);
            })
        ).toResolve('reject after 10 milliseconds should resolve'); // Fail
        Expect(
            new Promise((resolve, reject) => {
                setTimeout(() => {
                    resolve();
                }, 2);
            })
        ).toResolve('resolve after 2 milliseconds should resolve'); // Pass
    })
);

Output

Passed:  Expected [object Promise] ToResolve undefined.Description: resolve after 2 milliseconds should resolve
Failed:  Expected [object Promise] ToResolve undefined.Description: reject after 10 milliseconds should resolve
Passed:  Expected [object Promise] ToResolve undefined.Description: resolve after 20 milliseconds should resolve
Failed:  Expected [object Promise] ToResolve undefined.Description: reject after 100 milliseconds should resolve

[jamesadarich]

Hey @pathurs,

I'm not sure I quite understand this proposal. Is this a proposal to replace the way in which you write tests? i.e. a replacement for:

import { TestFixture, TestCase, Expect } from "alsatian";

@TestFixture("fixture")
export class NewTests {
   @TestCase(2)
   @TestCase(3)
   @TestCase(0)
   public testFunction(notOne: number) {
       Expect(1).not.toEqual(notOne);
   }
}

If so what's the reason for such a change and what benefits do you foresee? :)

[pathurs]

I've updated the original comment, hopefully it makes more sense.

[jamesadarich]

Indeed it does! Thanks for the clarification. I think for this is a great idea to try and coalesce all the new features we need into an architectural plan. There may be a few other considerations which may shape the architecture as well. Let me have a ponder and I'll add some more back here then we can get making it even more awesome!! :)

[jamesadarich]

My thoughts so far

Async as default

• ensure that the following try catch and handle as promise (even if sync)
  • setup
  • teardown
  • tests / test cases
• remove all "Async" versions of decorators
  • deprecate in v3.0.0
  • remove in v4.0.0

Expect rework

POC example to follow

• Expect is passed the built up test metadata so it can identify where it's being called from
• it identifies the current running test by checking stack using filepath to identify the fixture and function name to identify the test or testcase
  • note may need to run test cases synchronously to ensure we can identify which one is "current"
• failsWithMessage function can be called after a matcher e.g. `Expect(41).toBeLessThan(42).failsWithMessage("Expected input to be less than the answer");
• when expect is called this just creates a matcher instance that is added to the test case metadata to be executed once all expects have been added so we can identify if none have been added and warn

TAP compatibility

• need to ensure warnings show
• need to ensure multiple expects shown well on output

Considerations

• change in how we display failures at end of a run?
• only test fixtures should be run in parallel due to limitations on Expect rework and Setup and Teardown routines

[nebez]

Async as default makes a ton of sense. Having to discern between the two seems awkward and error-prone.