[Bug]: Using array of reporter types fails typescript validation: "Type 'string[]' is not assignable to type 'ReporterDescription'"

[asos-tomp]

Version

1.49.0

Steps to reproduce

Create a config with multiple reporters thus:

reporter: [["list"], ["html"]]

Expected behavior

With a javascript config, everything runs and multiple reports produced.

Actual behavior

[WebServer]     Argument of type '{ testDir: string; use: { baseURL: string; }; webServer: { reuseExistingServer: boolean; command: string; url: string; }; fullyParallel: boolean; forbidOnly: boolean; workers: number; reporter: string[][]; projects: { ...; }[]; }' is not assignable to parameter of type 'PlaywrightTestConfig<unknown, unknown>'.
[WebServer]       Types of property 'reporter' are incompatible.
[WebServer]         Type 'string[][]' is not assignable to type 'LiteralUnion<"html" | "line" | "github" | "list" | "dot" | "json" | "junit" | "null", string> | ReporterDescription[]'.
[WebServer]           Type 'string[][]' is not assignable to type 'ReporterDescription[]'.
[WebServer]             Type 'string[]' is not assignable to type 'ReporterDescription'.
[WebServer]               Type 'string[]' is not assignable to type 'readonly [string, any]'.
[WebServer]                 Target requires 2 element(s) but source may have fewer.

Additional context

No response

Environment

System:
    OS: macOS 14.7.1
    Memory: 3.91 GB / 32.00 GB
  Binaries:
    Node: 20.10.0 - ~/.nvm/versions/node/v20.10.0/bin/node
    Yarn: 1.22.22 - /opt/homebrew/bin/yarn
    npm: 10.2.3 - ~/.nvm/versions/node/v20.10.0/bin/npm
    pnpm: 9.12.2 - /opt/homebrew/bin/pnpm
  Languages:
    Bash: 3.2.57 - /bin/bash

[dgozman]

@asos-tomp Could you please share your tsconfig.json? Also, I am surprised that your logs are prefixed with [WebServer] - are you running tsc inside your web server?

[asos-tomp]

@asos-tomp Could you please share your tsconfig.json? Also, I am surprised that your logs are prefixed with [WebServer] - are you running tsc inside your web server?

This is running in a github actions workflow, using an npm script thus:

playwright test --config ./config.ts

...with that config thus:

import { devices, defineConfig } from "@playwright/test";

export default defineConfig({
  fullyParallel: true,
  forbidOnly: !!process.env.CI,
  workers: process.env.CI ? 1 : undefined,
  reporter: process.env.CI ? [["blob"], ["github"]] : [["list"], ["html"]],
  projects: [
    {
      name: "chromium",
      use: { ...devices["Desktop Chrome"] },
    },
  ],
  testDir: "./examples/next/src/app/fixtures",
  use: {
    baseURL: "http://localhost:3000/fixtures/"
  },
  webServer: {
    command: "npm run build && npm run start",
    url: "http://localhost:3000",
    reuseExistingServer: !process.env.CI,
  }
});

[dgozman]

@asos-tomp Could you still share your tsconfig.json please?

your logs are prefixed with [WebServer] - are you running tsc inside your web server?

It seems like the npm run build part of your webServer command performs the type checks. This is not something we see often - usually the web server is already built before you start testing, and the command is just npm run start.

[asos-tomp]

@asos-tomp Could you still share your tsconfig.json please?

{
  "compilerOptions": {
    "lib": [
      "dom",
      "dom.iterable",
      "esnext"
    ],
    "baseUrl": ".",
    "target": "es2018",
    "allowJs": true,
    "skipLibCheck": true,
    "strict": false,
    "noEmit": true,
    "incremental": true,
    "module": "esnext",
    "esModuleInterop": true,
    "moduleResolution": "node",
    "resolveJsonModule": true,
    "isolatedModules": true,
    "jsx": "preserve",
    "plugins": [
      {
        "name": "next"
      }
    ]
  },
  "include": [
    "next-env.d.ts",
    ".next/types/**/*.ts",
    "**/*.ts",
    "**/*.tsx",
    "playwright.config.ts"
  ],
  "exclude": [
    "node_modules"
  ]
}

your logs are prefixed with [WebServer] - are you running tsc inside your web server?

This is a vanilla Next.js example application, with bare bones configuration, containing the playwright config in the root. tsc may be running as part of the next setup, but by design its mostly opaque. There is a tsconfig.json as scaffolded by Next.

It seems like the npm run build part of your webServer command performs the type checks. This is not something we see often - usually the web server is already built before you start testing, and the command is just npm run start.

OK thanks.

I guess the fact the IDE is showing this error, irrespective of the contents of the webServer.command, is an indication that the types are incompatible with a documented use case? i.e. Does the principle apply that the reporters field of the config should allow string[][] as a type, as per the documentation, either way?

If I remove npm run build from the config, VSCode still shows:

 No overload matches this call.
  The last overload gave the following error.
    Argument of type '{ testDir: string; use: { baseURL: string; }; webServer: { reuseExistingServer: boolean; command: string; url: string; }; fullyParallel: boolean; forbidOnly: boolean; workers: number; reporter: string[][]; projects: { ...; }[]; }' is not assignable to parameter of type 'PlaywrightTestConfig<unknown, unknown>'.
      Types of property 'reporter' are incompatible.
        Type 'string[][]' is not assignable to type 'LiteralUnion<"github" | "list" | "html" | "dot" | "line" | "json" | "junit" | "null", string> | ReporterDescription[]'.
          Type 'string[][]' is not assignable to type 'ReporterDescription[]'.
            Type 'string[]' is not assignable to type 'ReporterDescription'.
              Type 'string[]' is not assignable to type 'readonly [string, any]'.
                Target requires 2 element(s) but source may have fewer.ts(2769)