Merge pull request #37008 from margelo/dev/improve-e2e-test-runner-su…

…ite-logging

[NoQA] dev: improve e2e tests DX

package.json

@@ -50,8 +50,8 @@
     "analyze-packages": "ANALYZE_BUNDLE=true webpack --config config/webpack/webpack.common.js --env envFile=.env.production",
     "symbolicate:android": "npx metro-symbolicate android/app/build/generated/sourcemaps/react/release/index.android.bundle.map",
     "symbolicate:ios": "npx metro-symbolicate main.jsbundle.map",
-    "test:e2e": "ts-node tests/e2e/testRunner.js --development --skipCheckout --skipInstallDeps --buildMode none",
-    "test:e2e:dev": "ts-node tests/e2e/testRunner.js --development --skipCheckout --config ./config.dev.js --buildMode skip --skipInstallDeps",
+    "test:e2e": "ts-node tests/e2e/testRunner.js --config ./config.local.ts",
+    "test:e2e:dev": "ts-node tests/e2e/testRunner.js --config ./config.dev.js",
     "gh-actions-unused-styles": "./.github/scripts/findUnusedKeys.sh",
     "workflow-test": "./workflow_tests/scripts/runWorkflowTests.sh",
     "workflow-test:generate": "ts-node workflow_tests/utils/preGenerateTest.js",

tests/e2e/ADDING_TESTS.md

@@ -1,51 +1,5 @@
 # Adding new E2E Tests
 
-## Running your new test in development mode
-
-Typically you'd run all the tests with `npm run test:e2e` on your machine.
-This will run the tests with some local settings, however that is not
-optimal when you add a new test for which you want to quickly test if it works, as the prior command
-still runs the release version of the app, which is hard to debug.
-
-I recommend doing the following.
-
-1. We need to compile a android development app version that has capturing metrics enabled:
-```bash
-# Make sure that your .env file is the one we need for e2e testing:
-cp ./tests/e2e/.env.e2e .env
-
-# Build the android app like you normally would with
-npm run android
-```
-2. Rename `./index.js` to `./appIndex.js`
-3. Create a new `./index.js` with the following content:
-```js
-require('./src/libs/E2E/reactNativeLaunchingTest');
-```
-4. In `./src/libs/E2E/reactNativeLaunchingTest.ts` change the main app import to the new `./appIndex.js` file:
-```diff
-- import '../../../index';
-+ import '../../../appIndex';
-```
-
-> [!WARNING]
-> Make sure to not commit these changes to the repository!
-
-Now you can start the metro bundler in e2e mode with:
-
-```bash
-CAPTURE_METRICS=true E2E_TESTING=true npm start -- --reset-cache
-```
-
-Then we can execute our test with:
-
-```
-npm run test:e2e:dev -- --includes "My new test name"
-```
-
-> - `--includes "MyTestName"` will only run the test with the name "MyTestName", but is optional
-
-
 ## Creating a new test
 
 Tests are executed on device, inside the app code.
@@ -144,8 +98,13 @@ Done! When you now start the test runner, your new test will be executed as well
 
 ## Quickly test your test
 
-To check your new test you can simply run `npm run test:e2e`, which uses the
-`--development` flag. This will run the tests on the branch you are currently on, runs fewer iterations and most importantly, it tries to reuse the existing APK and just patch into the new app bundle, instead of rebuilding the release app from scratch.
+> [!TIP]
+> You can only run a specific test by specifying the `--includes` flag:
+> ```sh
+> npm run test:e2e:dev -- --includes "My new test name"
+> ```
+
+It is recommended to run a debug build of the e2e tests first to iterate quickly on your test. Follow the explanation in the [README](./README.md) to create a debug build.
 
 ## Debugging your test
 

tests/e2e/README.md

@@ -6,44 +6,85 @@ run the actual app on a real device (physical or emulated).
 
 ![Example of a e2e test run](https://raw.githubusercontent.com/hannojg/expensify-app/5f945c25e2a0650753f47f3f541b984f4d114f6d/e2e/example.gif)
 
-To run the e2e tests:
-
-1. Connect an android device. The tests are currently designed to run only on android. It can be
-   a physical device or an emulator.
-
-2. Make sure Fastlane was initialized by running `bundle install`
-
-3. Run the tests with `npm run test:e2e`.
-  > 💡 Tip: To run the tests locally faster, and you are only making changes to JS, it's recommended to
-    build the app once with `npm run android-build-e2e` and from then on run the tests with
-    `npm run test:e2e -- --buildMode js-only`. This will only rebuild the JS code, and not the
-    whole native app!
-
-Ideally you want to run these tests on your branch before you want to merge your new feature to `main`.
 
 ## Available CLI options
 
 The tests can be run with the following CLI options:
 
 - `--config`: Extend/Overwrite the default config with your values, e.g. `--config config.local.ts`
 - `--includes`: Expects a string/regexp to filter the tests to run, e.g. `--includes "login|signup"`
-- `--skipInstallDeps`: Skips the `npm install` step, useful during development
-- `--development`: Applies some default configurations:
-  - Sets the config to `config.local.ts`, which executes the tests with fewer iterations 
-  - Runs the tests only on the current branch
-- `--buildMode`: There are three build modes, the default is `full`:
-  1. **full**: rebuilds the full native app in (e2e) release mode
-  2. **js-only**: only rebuilds the js bundle, and then re-packages
-                   the existing native app with the new package. If there
-                   is no existing native app, it will fallback to mode "full"
-  3. **skip**: does not rebuild anything, and just runs the existing native app
-- `--skipCheckout`: Won't checkout any baseline or comparison branch, and will just run the tests
-
-## Available environment variables
-
-The tests can be run with the following environment variables:
-
-- `baseline`: Change the baseline to run the tests again (default is `main`).
+
+## Running the tests on your machine
+
+You have two options when running the e2e tests:
+
+1. Run a debug build of the app (useful when developing a test)
+2. Run two (e2e) release builds against each other (useful to test performance regression and the suite as a whole)
+
+### Running a debug build
+
+1. You need to create a debug build of the app that's configured with some build flags to enable e2e testing.
+The build flags should be specified in your `./.env` file. You can use the `./tests/e2e/.env.e2e` file as a template:
+
+```sh
+cp ./tests/e2e/.env.e2e .env
+```
+
+> [!IMPORTANT]
+> There are some non-public environment variables that you still have to add to the `.env` file. Ask on slack for the values (cc @vit, @andrew, @hanno gödecke).
+
+2. Create a new android build like you usually would:
+
+```sh
+npm run android
+```
+
+3. We need to modify the app entry to point to the one for the tests. Therefore rename `./index.js` to `./appIndex.js` temporarily.
+
+4. Create a new `./index.js` with the following content:
+```js
+require('./src/libs/E2E/reactNativeLaunchingTest');
+```
+
+5. In `./src/libs/E2E/reactNativeLaunchingTest.ts` change the main app import to the new `./appIndex.js` file:
+```diff
+- import '../../../index';
++ import '../../../appIndex';
+```
+
+6. You can now run the tests. This command will invoke the test runner:
+
+```sh
+npm run test:e2e:dev
+```
+
+### Running two release builds
+
+The e2e tests are meant to detect performance regressions. For that we need to compare two builds against each other. On the CI system this is e.g. the latest release build (baseline) VS the currently merged PR (compare).
+
+You need to build the two apps first. Note that the two apps will be installed on the same device at the same time, so both apps have a different package name. Therefor, we have special build types for the e2e tests.
+
+1. Create a new android build for the baseline:
+
+> [!IMPORTANT]
+> There are some non-public environment variables that you still have to add to the `./tests/e2e/.env.e2e` and `./tests/e2e/.env.e2edelta` file. Ask on slack for the values (cc @vit, @andrew, @hanno gödecke).
+
+```sh
+npm run android-build-e2e
+```
+
+2. Create a new android build for the compare:
+
+```sh
+npm run android-build-e2edelta
+```
+
+3. Run the tests:
+
+```sh
+npm run test:e2e
+```
+
 
 ## Performance regression testing
 

tests/e2e/TestSpec.yml

@@ -22,7 +22,7 @@ phases:
     commands:
       - cd zip
       - npm install underscore ts-node typescript
-      - npx ts-node e2e/testRunner.js -- --skipInstallDeps --buildMode "skip" --skipCheckout --mainAppPath app-e2eRelease.apk --deltaAppPath app-e2edeltaRelease.apk
+      - npx ts-node e2e/testRunner.js -- --mainAppPath app-e2eRelease.apk --deltaAppPath app-e2edeltaRelease.apk
 
 artifacts:
   - $WORKING_DIRECTORY

tests/e2e/config.dev.js

@@ -7,4 +7,5 @@ export default {
     MAIN_APP_PATH: appPath,
     DELTA_APP_PATH: appPath,
     RUNS: 8,
+    BOOT_COOL_DOWN: 5 * 1000,
 };