Explore using a matrix in GitHub actions to call individual behave tags during E2E testing

[komish]

One of the biggest challenges we run into with E2E testing is with transient errors (e.g. GitHub rate limits). One test case failing requires us to re-run the entire E2E workflow because we currently call all BDD tests in a single call. For context, this is during the Workflow Test execution, where a release is taking place, and the full behave tag is being used.

Explore the use of GitHub's matrix functionality to call the full test suite as individual jobs. Explore concurrency, but keep in mind that we don't want the cluster to tumble because we're running multiple E2E test suites at once.

[komish]

I reconfigured testing for #359 to shorten the feedback loop, and it allowed some basic testing of matrix based execution. Here's the workflow file that I used.

name: "Matrix-based Testing"

on:
  workflow_dispatch:

jobs:
  get-features:
    runs-on: ubuntu-latest
    outputs:
      features: ${{ steps.find-features.outputs.features }}
    steps:
      - uses: actions/checkout@v4
      - name: find features
        id: find-features
        run: |
          cd tests/functional/behave_features
          echo features=$(find . -name '*.feature' | sed -e 's%\./%%g' | jq -R -s -c 'split("\n")[:-1]') | tee -a $GITHUB_OUTPUT
  run-tests:
    runs-on: ubuntu-latest
    needs: [get-features]
    strategy:
      fail-fast: false
      max-parallel: 4
      matrix: 
        feature-file: ${{ fromJson(needs.get-features.outputs.features) }}
    steps:
      - name: Checkout Base Branch
        uses: actions/checkout@v4
        with:
          token: ${{ secrets.BOT_TOKEN }}

      - name: Set up Python 3.x Part 1
        uses: actions/setup-python@v5
        with:
          python-version: "3.10"

      - name: Set up Python scripts
        run: |
          # set up python scripts
          echo "set up python script in $PWD"
          python3 -m venv ve1
          cd scripts
          ../ve1/bin/pip3 install -r requirements.txt
          ../ve1/bin/pip3 install .
          cd ..
      - name: Test CI Workflow
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          BOT_NAME: ${{ secrets.BOT_NAME }}
          BOT_TOKEN: ${{ secrets.BOT_TOKEN }}
          PR_BODY: "Triggered by ${{ github.event.sender.html_url }} from ${{ github.event.repository.html_url }}"
        run: |
          ve1/bin/behave tests/functional/behave_features/ --include ${{ matrix.feature-file }} --tags=full --logging-level=WARNING --no-capture --no-color

This isn't exactly how we would want to run this testing because I was running it against a branch (main in my case) with a workflow_dispatch, but it gives an example of how this might work.

Notable things:

• I set max_parallel to 4 because I hit short-term API limits if I used more than that. I think it was the number of API calls per hour or per minute. Can't remember off-hand.

• We find "feature" files using system primitives (just the find command running in the right directory). We also use a tag - so in effect we're saying "scenarios for this feature with that tag".

• fail-fast is disabled so we can just let all the tests that will pass do so, and then just re-run the ones that fail in case they fail in a transient way (e.g. GitHub API status is down).