Testing.md

Testing

Our team is composed by multiple squads which are responsible for specific parts of the product.

Based on that, each squad is responsible for owning and maintaining their own testing, from unit tests to UI Automation tests, meaning that any failure should be investigated and fixed by the squad responsible.   If the squad is unavailable, e.g. holidays, then one of the Release Engineers should take ownership of the failure when we are in the process of releasing a new version of that app; otherwise it should fallback to the Support Engineer.

When a UI test lane fails, the responsible squad is notified on Slack.

Execution

We use fastlane to run most of our common and frequent tasks and UI Automation tests are no exception. We have specific lanes defined for each part of the product and they can be triggered on demand, either by running them locally, or on CI from the #ios-build channel.

# Trigger a lane to be executed locally from the terminal
$ bundle exec fastlane {lane}

# Trigger a lane to be executed on CI from the #ios-build channel
/fastlane {lane} branch:{branch}

Listing all lanes available can be done with bundle exec fastlane list in the terminal. More details on Slack integration with CircleCI can be found here.

• UI Test lanes all start with ui_test_* and are run nightly. The list of available tests can be found in fastlane/Lanes/uitests.
• The ui_test_babylon lane is special, allowing to test a specific feature on demand using fastlane ui_test_babylon feature:MyAwesomeFeature using the XCTest class name for the feature: parameter.

QA

Each squad has a QA Engineer responsible for testing all the work done by the squad. They should work together with the engineers to make sure we are delivering good and stable features.

Our UI Automation tests take a significant amount of time to run so it's only executed overnight. This means we don't detect if some feature breaks with the changes of a pull request automatically. For this reason, both the iOS and QA Engineers are responsible for checking any potential failure in any change made within the squad. This can be achieved by triggering the relevant lane for testing and ideally, this should be done during the review of the pull request, to avoid breakage.

When there's a suspicion that the changes may affect other parts of the product, the relevant lanes should be triggered to be tested, or ultimately, the entire suite should be executed to assert that nothing breaks within the product. This can be identified and done by either the iOS Engineer or the QA Engineer.

This also shows how fundamental it is having good communication between the iOS Engineer and the QA Engineer so they can work together effectively and be always in sync.