Only uploading single report for multiple runs

[dmattia]

I am attempting to run multiple tests on a site and get a summary of the performance across all runs. I am seeing output indicating that multiple runs happen, and when I view the stored artifacts on Github actions, I see multiple .html reports, but the uploaded reports (like https://storage.googleapis.com/lighthouse-infrastructure.appspot.com/reports/1666985268269-5098.report.html) are just from one of the three runs, not a summary across all runs.

Is this expected?

To Reproduce

I have a basic CI workflow that looks like:

name: Perf testing

on:
  pull_request:

jobs:
  test:
    runs-on: ubuntu-22.04
    timeout-minutes: 10
    steps:
      - uses: actions/checkout@v2
      - uses: actions/[email protected]
        with:
          node-version: '14.17.3'
      - name: Run Lighthouse on urls with lighthouserc
        uses: treosh/lighthouse-ci-action@v9
        with:
          urls: '<some_url>'
          uploadArtifacts: true
          temporaryPublicStorage: true
          runs: 3
          configPath: '${{ github.workspace }}/lighthouse/lighthouserc.json'

The referenced lighthouserc.json contains:

{
  "ci": {
    "collect": {
      "numberOfRuns": 3,
      "settings": {
        "configPath": "./lighthouse/lighthouse-config.js"
      }
    }
  }
}

and the config file contains:

module.exports = {
  extends: 'lighthouse:default',
  settings: {
    emulatedFormFactor: 'desktop',
    onlyCategories: ['performance'],
    throttlingMethod: 'provided',
    // You can find audit names here:
    // https://github.com/GoogleChrome/lighthouse/blob/eba2a4d19c5786dc37e993858ff4b663181f81e5/lighthouse-core/config/default-config.js#L174
    onlyAudits: [
      'metrics/first-contentful-paint',
      'metrics/first-meaningful-paint',
      'metrics/speed-index',
      'critical-request-chains',
    ],
  },
};