Skip to content

Add Qa seeder for groups & friends #38

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 30 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
9630e60
chore: add tooling v1
Bilb Apr 19, 2025
ba37403
chore: update Avatar color test to use prebuiltstate
Bilb Apr 20, 2025
0133949
fix: updated some more tests
Bilb Apr 20, 2025
8489030
fix: moving "Disappear after send groups" to prebuilt setup
Bilb Apr 20, 2025
f891cc7
chore: replaced all createGroup through UI that I easily could
Bilb Apr 20, 2025
085db8e
fix: made the platform be a tag instead of just a string
Bilb Apr 20, 2025
6430730
fix: replaced a fair bit of newContact with state equivalent
Bilb Apr 20, 2025
f1b1fab
chore: big refactor of bothPlatformsIt
Bilb Apr 21, 2025
2de1e3f
feat: assign each test a countOfDevice required
Bilb Apr 24, 2025
6397f8f
feat: allow to run count-tagged tests on multiple workers
Bilb Apr 25, 2025
6e968cc
feat: add state with one user linked to 2 devices
Bilb Apr 25, 2025
3f31a08
chore: made focusGroup/FriendsConvo mandatory
Bilb Apr 25, 2025
60b2e95
chore: run all the high risk suite to grab new perf
Bilb Apr 25, 2025
03e96dd
feat: move "Delete message linked device" to state
Bilb Apr 26, 2025
8a2db21
feat: add a csv output section
Bilb Apr 26, 2025
68a3e59
chore: run the full suite to grab csv stats
Bilb Apr 26, 2025
68564fa
chore: major refactor from device1/2/3 to alice1/bob1 etc
Bilb Apr 26, 2025
b9a9f21
feat: if emulators are not running for tests, fail
Bilb Apr 26, 2025
f47204d
chore: clean up
Bilb Apr 26, 2025
0ab9332
fix: file upload was linked to single device
Apr 26, 2025
0d26a9e
fix: sendCommunityInviteMessageAndroid
Apr 27, 2025
3433a13
fix: fixed a few more tests and added csv result write
Apr 27, 2025
25445ec
feat: upload test results (diff img) as artefacts
Apr 27, 2025
9652816
chore: move sessionreporter to a package
Bilb Apr 27, 2025
e56a913
Merge branch 'add-tooling' of github.com:session-foundation/session-a…
Bilb Apr 27, 2025
0a46059
chore: break down big workflow file into composite actions
Bilb Apr 27, 2025
b84112d
chore: remove need to have two levels of fork
Bilb Apr 27, 2025
affd448
fix: use yarn 4.1.1
Bilb Apr 27, 2025
97866d9
chore: add print runner details shared action
Bilb Apr 27, 2025
242e5ee
chore: reuse shared action for ios
Bilb Apr 27, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
Expand Up @@ -2,3 +2,5 @@ run/screenshots/android/landingpage_new_account.png filter=lfs diff=lfs merge=lf
run/screenshots/android/landingpage_restore_account.png filter=lfs diff=lfs merge=lfs -text
run/screenshots/ios/landingpage_new_account.png filter=lfs diff=lfs merge=lfs -text
run/screenshots/ios/landingpage_restore_account.png filter=lfs diff=lfs merge=lfs -text
.yarn/releases/yarn-4.1.1.cjs filter=lfs diff=lfs merge=lfs -text
.yarn/releases/yarn-*.cjs filter=lfs diff=lfs merge=lfs -text
223 changes: 96 additions & 127 deletions .github/workflows/android-regression.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
name: Android regression tests
run-name: '${{ inputs.RISK }} regressions on: ${{ inputs.BRANCH_TO_CHECKOUT }} (${{ inputs.APPIUM_REPO }})'
run-name: '${{ inputs.RISK }} regressions on ${{ github.head_ref || github.ref }}'

on:
workflow_dispatch:
Expand All @@ -8,7 +8,7 @@ on:
description: 'url to test'
required: true
type: string
default: https://oxen.rocks/session-foundation/session-android/release/1.20.3/session-android-20241101T052421Z-be16d3bf9-universal.tar.xz
default: https://oxen.rocks/session-foundation/session-android/release/1.22.1/session-android-20250406T231441Z-a9bfe783a-universal.tar.xz

RISK:
description: 'risks to target'
Expand All @@ -20,45 +20,6 @@ on:
- 'low-risk'
- ''

APPIUM_REPO:
description: 'appium repo to checkout'
required: true
type: choice
options:
- session-foundation/session-appium
- burtonemily/session-appium
- Miki-Session/session-appium
- bilb/session-appium
default: session-foundation/session-appium

BRANCH_TO_CHECKOUT:
description: 'branch to checkout'
required: true
type: string
default: test-ci-regression

# SHARD_NUMBER:
# description: 'shard number'
# required: true
# type: choice
# options:
# - '1'
# - '2'
# - '3'
# - '4'
# default: '1'

# SHARD_COUNT:
# description: 'shard count'
# required: true
# type: choice
# options:
# - '1'
# - '2'
# - '3'
# - '4'
# default: '1'

PLAYWRIGHT_RETRIES_COUNT:
description: 'retries of failing tests to do at most'
required: true
Expand All @@ -85,108 +46,64 @@ on:
- '0'
- '1'
default: '0'
PLAYWRIGHT_WORKERS_COUNT:
description: 'number of workers to use - ONLY 1 WORKER IS SUPPORTED FOR ANDROID'
required: true
type: choice
options:
- '1'
default: '1'

# concurrency:
# group: ${{ github.workflow }}
# cancel-in-progress: true

jobs:
android-regression:
runs-on: [self-hosted, linux, X64, qa-android]
env:
IOS_APP_PATH_PREFIX: '../extracted/Session.app'
ANDROID_APK: '../extracted/session-android.apk'
PLATFORM: 'android'
CI: 1
IOS_APP_PATH_PREFIX: './extracted/Session.app'
ANDROID_APK: './extracted/session-android.apk'
APPIUM_ADB_FULL_PATH: '/opt/android/platform-tools/adb'
ANDROID_SDK_ROOT: '/opt/android'
PLAYWRIGHT_RETRIES_COUNT: ${{ github.event.inputs.PLAYWRIGHT_RETRIES_COUNT }}
PRINT_FAILED_TEST_LOGS: ${{ github.event.inputs.PRINT_FAILED_TEST_LOGS }}
PRINT_ONGOING_TEST_LOGS: ${{ github.event.inputs.PRINT_ONGOING_TEST_LOGS }}
IOS_FIRST_SIMULATOR: '49651A15-3E14-4BAF-8B8E-0C630C35B8DD'
IOS_SECOND_SIMULATOR: '7CA21E47-56AC-4B76-82A4-1D5199515F38'
IOS_THIRD_SIMULATOR: 'FE1DEDC7-D530-41E7-9B38-86ED2DDB97CD'
IOS_FOURTH_SIMULATOR: '561C621C-2CEF-4FCC-87B7-E6221718D15C'
IOS_FIFTH_SIMULATOR: '5AA87E03-5831-4209-87A0-21E5ECDC490F'
IOS_SIXTH_SIMULATOR: '6FB10AF3-3970-46BA-B414-1206AA4E726D'
IOS_SEVENTH_SIMULATOR: '0BDA36C0-83F4-46DF-ADC9-FF30E969D4BD'
IOS_EIGHTH_SIMULATOR: '1C61A0B6-3085-46B6-9B2E-6E9A1B594944'
IOS_NINTH_SIMULATOR: '7C69D129-BD40-4BC0-9A0D-6A7E990F23E8'
IOS_TENTH_SIMULATOR: '275E0C5B-2458-4490-831D-4CE86C8EC704'
IOS_ELEVENTH_SIMULATOR: 'EA5C5BCB-1E3E-4FFC-88BB-8D1DE354BC3B'
IOS_TWELFTH_SIMULATOR: '41514F87-DC49-4C81-B60C-03D1F8653A42'

# echo "SHARD_NUMBER ${{ github.event.inputs.SHARD_NUMBER }}"
# echo "SHARD_COUNT ${{ github.event.inputs.SHARD_COUNT }}"
SDK_MANAGER_FULL_PATH: '/Users/emilyburton/Library/Android/sdk/cmdline-tools/latest/bin/sdkmanager'
AVD_MANAGER_FULL_PATH: '/Users/emilyburton/Library/Android/sdk/cmdline-tools/latest/bin/avdmanager'
ANDROID_SYSTEM_IMAGE: 'system-images;android-35;google_atd;x86_64'
EMULATOR_FULL_PATH: '/Users/emilyburton/Library/Android/sdk/emulator/emulator'
IOS_1_SIMULATOR: '<just_not_empty>'
IOS_2_SIMULATOR: '<just_not_empty>'
IOS_3_SIMULATOR: '<just_not_empty>'
IOS_4_SIMULATOR: '<just_not_empty>'
IOS_5_SIMULATOR: '<just_not_empty>'
IOS_6_SIMULATOR: '<just_not_empty>'
IOS_7_SIMULATOR: '<just_not_empty>'
IOS_8_SIMULATOR: '<just_not_empty>'
IOS_9_SIMULATOR: '<just_not_empty>'
IOS_10_SIMULATOR: '<just_not_empty>'
IOS_11_SIMULATOR: '<just_not_empty>'
IOS_12_SIMULATOR: '<just_not_empty>'

steps:
- uses: actions/checkout@v4
- name: Runner Details
run: |
echo "BRANCH_TO_CHECKOUT ${{ github.event.inputs.BRANCH_TO_CHECKOUT }}"
echo "APPIUM_REPO ${{ github.event.inputs.APPIUM_REPO }}"
echo "APK_URL ${{ github.event.inputs.APK_URL }}"
echo "RISK ${{ github.event.inputs.RISK }}"
with:
lfs: true

- uses: ./github/actions/print-runner-details
with:
APK_URL: ${{ github.event.inputs.APK_URL }}
RISK: ${{ github.event.inputs.RISK }}
PRINT_FAILED_TEST_LOGS: ${{ github.event.inputs.PRINT_FAILED_TEST_LOGS }}
PRINT_ONGOING_TEST_LOGS: ${{ github.event.inputs.PRINT_ONGOING_TEST_LOGS }}
PLAYWRIGHT_RETRIES_COUNT: ${{ github.event.inputs.PLAYWRIGHT_RETRIES_COUNT }}

- name: Download APK
- name: Download APK & extract it
run: |
pwd
wget -q -O session-android.apk.tar.xz ${{ github.event.inputs.APK_URL }}
ls
pwd

- name: Extract APK
run: |
tar xf session-android.apk.tar.xz
pwd

- name: Rename extracted folder
run: |
ls
mv session-android-*universal extracted
ls extracted
mv extracted/*.apk extracted/session-android.apk
ls extracted
pwd

- uses: actions/checkout@v4
with:
repository: ${{ github.event.inputs.APPIUM_REPO }}
ref: ${{ github.event.inputs.BRANCH_TO_CHECKOUT }}
path: 'forked-session-appium'
lfs: true

- uses: actions/setup-node@v4
with:
# node-version-file: 'forked-session-appium/.nvmrc'
node-version: 18.15.0

- name: Install yarn
run: |
npm install -g yarn
corepack enable
yarn set version 4.1.1

- name: Install test dependencies
run: |
cd forked-session-appium
ls
git status
touch yarn.lock
yarn install --immutable

- name: Start Appium server
run: |
cd forked-session-appium
yarn "./node_modules/.bin/appium server --use-drivers=uiautomator2,xcuitest --port 8110 --allow-cors"

- name: Build the Android tests
run: |
cd forked-session-appium
yarn tsc
- name: Setup for both platforms
uses: ./github/actions/setup

- name: Restart adb server
shell: bash
Expand All @@ -202,19 +119,71 @@ jobs:
source ./scripts/ci.sh
start_with_snapshots

- name: List all tests
- name: List tests part of this run
uses: ./github/actions/list-tests
with:
PLATFORM: ${{ env.PLATFORM }}
RISK: ${{ github.event.inputs.RISK }}

- name: Run the 1-devices tests ​​with 4 workers
continue-on-error: true
id: devices-1-test-run
env:
PLAYWRIGHT_WORKERS_COUNT: 4
DEVICES_PER_TEST_COUNT: 1
run: |
pwd
npx playwright test --grep "(?=.*@${PLATFORM})(?=.*@${DEVICES_PER_TEST_COUNT}-devices)(?=.*@${{ github.event.inputs.RISK }})" #Note: this has to be double quotes

- name: Upload results of this run
uses: ./github/actions/upload-test-results
with:
PLATFORM: ${{ env.PLATFORM }}
UPLOAD_IDENTIFIER: 'devices-1-test-run'

- name: Run the 2-devices tests ​​with 2 workers
continue-on-error: true
id: devices-2-test-run
env:
PLAYWRIGHT_WORKERS_COUNT: 2
DEVICES_PER_TEST_COUNT: 2
run: |
cd forked-session-appium
pwd
npx playwright test --list --grep "@${{ github.event.inputs.RISK }}"
npx playwright test --grep "(?=.*@${PLATFORM})(?=.*@${DEVICES_PER_TEST_COUNT}-devices)(?=.*@${{ github.event.inputs.RISK }})" #Note: this has to be double quotes

- name: Run the tests # ${{ github.event.inputs.SHARD_NUMBER }}​​/${{ github.event.inputs.SHARD_COUNT }}​​
- name: Upload results of this run
uses: ./github/actions/upload-test-results
with:
PLATFORM: ${{ env.PLATFORM }}
UPLOAD_IDENTIFIER: 'devices-2-test-run'

- name: Run the tests ​​other tests with 1 worker
continue-on-error: true
id: other-devices-test-run
env:
PLAYWRIGHT_WORKERS_COUNT: 1
DEVICES_PER_TEST_COUNT: 4
run: |
cd forked-session-appium
pwd
yarn test 'android "@${{ github.event.inputs.RISK }}"'
npx playwright test --grep "(?=.*@${PLATFORM})(?=.*@${{ github.event.inputs.RISK }})" --grep-invert "@1-devices|@2-devices" #Note: this has to be double quotes

- name: Upload results of this run
uses: ./github/actions/upload-test-results
with:
PLATFORM: ${{ env.PLATFORM }}
UPLOAD_IDENTIFIER: 'devices-other-test-run'

- name: Upload csv of this whole run
uses: ./github/actions/upload-csv-test-results
if: always()
with:
PLATFORM: ${{ env.PLATFORM }}

- name: Check if any tests failed
if: steps.devices-1-test-run != 'success' || steps.devices-2-test-run != 'success' || steps.other-devices-test-run.outcome != 'success'
run: echo 'Some test failed, see above'; exit 1

- name: Kill all running emulators
- name: Stop emulators
if: always()
continue-on-error: true # just so we don't fail
shell: bash
Expand Down
Loading
Loading