Implement page.on('request')

[ankur22]

Feature Description

What

We want to implement page.on('request'). More details of the API can be found here. The request object that is returned to the handler is detailed here. This is a read only API, and no modifications can be done on the request object. k6 browser should not wait for a response from the handler, it fire and forgets.

Why

Users have asked to be able to read the requests that are being sent out from chromium to the website under test. This can be helpful in validating/asserting certain aspects of the request which could be important for the test to be deemed successful.

Comments

While testing the Playwright implementation i noticed that some of the methods on the request object are async. An example of this is data.response():

    page.on('request', async (data) => {
      console.log("url: " + data.url());
      if (data.url() === "https://quickpizza.grafana.com/api/quotes") {
        console.log(await data.response());
      }
    })

What's interesting about this is that the call to data.response() will in fact wait until a response is received. The current implementation of response on the request object does not fulfil this behaviour and we will need to change that too.

Async methods on request that will need to be checked to see if they behave the same as in PW:

• allHeaders
• headerValue
• headersArray
• response
• sizes

Suggested Solution (optional)

No response

Already existing or connected issues / PRs (optional)

No response

[ankur22]

I've started work on implementing this API here: #4290

I have tested the change with:

import { browser } from 'k6/browser'

export const options = {
  scenarios: {
    ui: {
      executor: 'shared-iterations',
      options: {
        browser: {
          type: 'chromium',
        },
      },
    },
  },
}

export default async function () {
  const page = await browser.newPage()

  const requestCounter = (function() {
    let count = 0;
    return () => ++count;
  })();

  page.on('request', async (request) => {
    const currentCount = requestCounter();
    console.log(JSON.stringify({
      requestNumber: currentCount,
      allHeaders: await request.allHeaders(),
      frameUrl: request.frame().url(),
      acceptLanguageHeader: await request.headerValue('Accept-Language'),
      headers: request.headers(),
      headersArray: await request.headersArray(),
      isNavigationRequest: request.isNavigationRequest(),
      method: request.method(),
      postData: request.postData(),
      postDataBuffer: request.postDataBuffer() ? String.fromCharCode.apply(null, new Uint8Array(request.postDataBuffer())) : null,
      resourceType: request.resourceType(),
      response: await request.response(),
      size: request.size(),
      timing: request.timing(),
      url: request.url()
    }, null, 2));
  })

  await page.goto('https://quickpizza.grafana.com/', { waitUntil: 'networkidle' })

  await page.locator('//button[text()="Pizza, Please!"]').click();

  await page.waitForTimeout(1000);

  await page.close();
}

An example of one of the console logs:

INFO[0002] {
  "requestNumber": 23,
  "allHeaders": {
    "content-type": "text/plain;charset=UTF-8",
    "authorization": "Token I7A7HPOh6vIEx1dx",
    "referer": "https://quickpizza.grafana.com/",
    "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36",
    "accept-language": "en-US"
  },
  "frameUrl": "https://quickpizza.grafana.com/",
  "acceptLanguageHeader": "en-US",
  "headers": {
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36",
    "Accept-Language": "en-US",
    "Content-Type": "text/plain;charset=UTF-8",
    "Authorization": "Token I7A7HPOh6vIEx1dx",
    "Referer": "https://quickpizza.grafana.com/"
  },
  "headersArray": [
    {
      "name": "User-Agent",
      "value": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36"
    },
    {
      "name": "Accept-Language",
      "value": "en-US"
    },
    {
      "name": "Content-Type",
      "value": "text/plain;charset=UTF-8"
    },
    {
      "name": "Authorization",
      "value": "Token I7A7HPOh6vIEx1dx"
    },
    {
      "name": "Referer",
      "value": "https://quickpizza.grafana.com/"
    }
  ],
  "isNavigationRequest": false,
  "method": "POST",
  "postData": "{\"maxCaloriesPerSlice\":1000,\"mustBeVegetarian\":false,\"excludedIngredients\":[],\"excludedTools\":[],\"maxNumberOfToppings\":5,\"minNumberOfToppings\":2}",
  "postDataBuffer": "{\"maxCaloriesPerSlice\":1000,\"mustBeVegetarian\":false,\"excludedIngredients\":[],\"excludedTools\":[],\"maxNumberOfToppings\":5,\"minNumberOfToppings\":2}",
  "resourceType": "Fetch",
  "response": null,
  "size": {
    "headers": 302,
    "body": 145
  },
  "timing": null,
  "url": "https://quickpizza.grafana.com/api/pizza"
}  source=console

Differences

There are some differences between k6 browser and Playwright that are worth thinking about before going ahead with continuing on with the implementation.

Some of the methods rely on being able to wait for the request to be sent and the response to be returned, which can then be accessed from the request:

• request.response() -- This should wait for the response object to be created and set. At the moment it returns null.
• request.size() -- This has three issues.
  • Firstly in Playwright it is sizes().
  • Secondly it is sync and not async.
  • Finally, the object definition is different. The k6 implementation only takes into account the request body and header size. The Playwright implementation takes into the request and response. IMO, if we wanted to match with Playwright, then we should create a new method sizes and deprecate size.
• request.timing() -- Like with request.response, this should wait for the response to be available to then be able to calculate the timings.
• headers -- we currently lack the support of getting the security based headers. We have an open issue to implement this: Response.AllHeaders() does not include security-related headers #4291.

Alternate

Instead of matching Playwright 1:1, we could polish off what we have, and document only the methods that work. When it comes to implementing page.on('response') all the methods that aren't in the request object from page.on('request') should be available in the response object from page.on('response'), including the original request with response.request().

I think it is odd that we have page.on('request') which can also be used to retrieve the response and anything else that is derived from that object. If a user wants to work with the response, then they should ideally work with page.on('response').

In summary, it's not clear to me why Playwright chose to be able to work with the response when working with page.on('request').