CI Pipeline through GH Actions

[nicolasbisurgi]

Adding automated testing through GitHub Actions. This is in response to #1145

[MariusWirtz]

Nice work @nicolasbisurgi
This should improve the robustness of our releases and provide transparency when new contributions break existing features.
Especially running tests automated and in parallel against v11 and v12 should make our lifes easier.

@onefloid
any thoughts on this one?

[onefloid]

Nice work @nicolasbisurgi It looks very promising. I can't test it at the moment because I have got no internet facing tm1 server available. @MariusWirtz was very quick. However I have left some comments on the code.

[onefloid]

Not for now, because the tests are not yet ready for this, but for the future: it would be good to have an option whether you want to run the tests in parallel or not.

[nicolasbisurgi]

It is actually working like that right now, you can define that in the second parameter when you run the workflow:

In my case I have 3 environments defined:

• tm1-11-onprem: a v11 instance we have running in an EC2 instance
• tm1-11-cloud: a v11 instance hosted in IBM PA Cloud
• tm1-12-mcsp: a v12 instance running as SaaS in AWS.

If I put those three, GH Actions will create 3 different and independent runners on the backend that run in parallel. Each of them will have its own config.ini file showing the TM1_CONNECTION and TM1_CONNECTION_SECRET from its own environment.

Below is a chart explaining the flow:

graph TD
    A[Workflow Dispatch] --> B[Input: Environments]
    B --> C[Parse Environments JSON]
    C --> D{Matrix Strategy}

    %% Parallel jobs for each environment
    D -->|tm1-11-onprem| E1
    D -->|tm1-11-cloud| E2
    D -->|tm1-12-mcsp| E3

    %% Independent Runner for tm1-11-onprem
    subgraph Runner: tm1-11-onprem
        E1[Job: Test tm1-11-onprem] --> F1[Checkout Code]
        F1 --> G1[Set Up Python]
        G1 --> H1[Install Dependencies]
        H1 --> I1[Retrieve TM1 Connection Details]
        I1 --> J1[Generate config.ini]
        J1 --> K1[Run Tests]
    end

    %% Independent Runner for tm1-11-cloud
    subgraph Runner: tm1-11-cloud
        E2[Job: Test tm1-11-cloud] --> F2[Checkout Code]
        F2 --> G2[Set Up Python]
        G2 --> H2[Install Dependencies]
        H2 --> I2[Retrieve TM1 Connection Details]
        I2 --> J2[Generate config.ini]
        J2 --> K2[Run Tests]
    end

    %% Independent Runner for tm1-12-mcsp
    subgraph Runner: tm1-12-mcsp
        E3[Job: Test tm1-12-mcsp] --> F3[Checkout Code]
        F3 --> G3[Set Up Python]
        G3 --> H3[Install Dependencies]
        H3 --> I3[Retrieve TM1 Connection Details]
        I3 --> J3[Generate config.ini]
        J3 --> K3[Run Tests]
    end

Loading

[onefloid]

No I mean that the tests itself run in parallel with, e.g. pytest -n auto .\Tests\ElementService_test.py

Compare #1105

[nicolasbisurgi]

Oh! I see, so you re-worked the ElementServce_test.py to be able to run independently from others?
Is this something we can do (or is already done) for other unit test cases? Or you are suggesting to run this one independently and the others sequentially?
Please feel free to create a PR against the yml and put this change in

[onefloid]

Sorry for being so unprecise. The clue is the -n option of pytest starts 1..n workers.
At the moment only a few tests can run with more than one worker. But in future it would be beneficial to specify how many workers should run in the pipeline.

[MariusWirtz]

Running tests in parallel is beneficial when you're working locally and want quick feedback IMO.

However, I'm concerned that parallel tests might occasionally fail due to potential race conditions or non-deterministic behavior in TM1. In the CI pipeline, I would prefer certainty over performance.

On the desktop, where you can easily re-run a failed test, I would prioritize performance over certainty.

[MariusWirtz]

With the "Matrix Strategy" we should also get benefits of parallelization but the parallel executions should be isolated from one another.