targets

Targets definitions

This directory contains target definitions used for configuring the targets and tests that will be executed by builders in the chromium project that use the chromium_tests recipe module for executing tests. This includes most builders that build and run tests. Builders running PRESUBMIT.py tests and builders running fuzz tests are not included.

Files

• basic_suites.star - This contains definitions of basic suites, which specify tests to include as well as optionally specify various details for those tests such as which test target to actually build and run, additional args that should be used or default swarming details. These suites can be referenced by compound suites in compound_suites.star, matrix compound suites in matrix_compound_suites.star or set as a test suite for a builder in waterfalls.pyl.

• binaries.star - This contains declarations with binary mappings used when declaring tests. The declarations map the GN target name to the ninja label that needs to be built to produce the actual executable for the test. Additional details can be specified as part of the declaration that will be used as part of the process of generating target spec files. Tests defined using targets.tests.gtest_test and target.tests.isolated_script_test all require a binary to be defined, which is set via the binary field. If the binary field is not set, then there must be a binary declared that has the same name as the test. Tests defined using targets.tests.junit_test implicitly define a binary that results in an entry in the generated gn_isolate_map.pyl file.

  Available binary types are:

  • console_test_launcher
  • generated_script
  • script
  • windowed_test_launcher

• bundles.star - This contains declarations of bundles, which are groupings of tests and compile targets that can be referenced by builders that set their tests in starlark. These bundles cannot be referenced by any legacy suites or by builders setting their tests in waterfalls.pyl.

• compile_targets.star - This contains definition of compile-only targets. These targets can only be built by a builder and have no provision for execution. These targets can be included in the additional_compile_targets field for a builder in waterfalls.pyl.

• compound_suites.star - This contains definitions of compound suites, which specify basic suites to collect into a larger suite. These suites can set as a test suite for a builder in waterfalls.pyl.

• matrix_compound_suites.star - This contains definitions of matrix compound suites, which specify basic suites to collect into a larger suite, optionally applying mixins to all the tests of a basic suite and/or expanding tests in a basic suite with one or more variants. For each suite where variants are specified, each test in the suite will generate a test spec for each variant with the variant applying modifications to the generated test spec. These suites can be set as a test suite for a builder in waterfalls.pyl.

• mixins.star - This contains definitions of mixins that can be used to modify the generated test specs. These mixins can be specified when defining the details of a test in a basic suite to modify that test, when configuring a basic suite in a matrix compound suite to modify all tests in that expansion of the basic suite or on a builder in waterfalls.pyl to modify all tests for the builder.

• tests.star - This contains definitions of test targets. These targets can be referenced in basic suites. Additionally, they can contain details such as args or mixins. These details will be transferred to the generated test_suites.pyl since //testing/buildbot doesn't use separate target declarations for tests.

  Available test types are:

  • gtest_test
  • gpu_telemetry_test
  • isolated_script_test
  • junit_test
  • script_test

• variants.star - This contains definitions of variants, which allow for expanding a single test definition into multiple generated test specs and apply additional modifications to the generated test specs. They can be specified when configuring a basic suite in a matrix compound suite to expand all of the tests in the basic suite with each of the specified variants.

  • cros-skylab-variants.json, lacros-version-skew-variants.json - These files are consumed by variants.star to define certain subsets of variants. Json files are used because they are auto-generated by builders running periodically and json is easily manipulated programatticaly.

Tests in starlark

Currently, a migration is in process to enable tests for a builder to be configured as part of the builder definition itself. In order to avoid having to manually sync definitions in two locations/systems, some of the .pyl that used to be hand-written files in //testing/buildbot are now generated from the starlark definitions. Due to intentional design decisions in lucicfg, the files generated from the //infra/config starlark can't be located outside of //infra/config/generated. //testing/buildbot/generate_buildbot_json.py has been updated to read gn_isolate_map.pyl, mixins.pyl, test_suites.pyl and variants.pyl from //infra/config/generated/testing.

Updating these .pyl files requires the following process:

1. Modify starlark files

   .pyl file	starlark file to edit
   gn_isolate_map.pyl	binaries.star (or tests.star for test defined using targets.test.junit_test)
   mixins.pyl	mixin.star
   variants.pyl	variants.star
   test_suites.pyl (basic_suites entries)	basic_suites.star (or tests.star/binaries.star or for values that apply to all instances of a test/binary)
   test_suites.pyl (compound_suites entries)	compound_suites.star
   test_suites.pyl (matrix_compound_suites entries)	matrix_compound_suites.star

2. Regenerate the config by running main.star

   lucicfg infra/config/main.star

   On mac or linux, you can just do: infra/config/main.star

Then you can make any edits you wish to the hand-written .pyl files in //testing/buildbot (waterfalls.pyl and test_suite_exceptions.pyl) and run generate_buildbot_json.py to generate the targets spec files.

Due to angle using mixins.pyl via a subtree repo that exports //testing/buildbot, if mixins.pyl was modified by the above steps, it's necessary to sync that file to //testing/buildbot.

1. Copy mixins.pyl from //infra/config/generated/testing to //testing/buildbot by running sync-pyl-files.py

   infra/config/scripts/sync-pyl-files.py

Setting tests for a builder in starlark

It is now possible to specify test in starlark for builders in limited conditions:

• No legacy matrix compound suites can be referenced
• Cannot rely on any fields being set on the builder's waterfall in waterfalls.pyl except mixins
• Cannot rely on any fields besides additional_compile_targets, test_suites, mixins, os_type and use_swarming being set for the builder in waterfalls.pyl (fields that are present in targets.mixin can be specified by specifying an in-place mixin in the mixins field instead)
  • Under test_suites, test can only be specified for the scripts, gtest_tests and isolated_scripts keys
    • Only the following fields can be set on gtests & isolated scripts either directly or via mixins:
      • args
      • android_args (expanding these isn't supported yet, so this won't take effect, but won't be rejected)
      • android_swarming (expanding these isn't supported yet, so this won't take effect, but won't be rejected)
      • ci_only
      • isolate_profile_data
      • merge
      • precommit_args
      • resultdb
      • swarming
      • test (only allowed in declaration of gtest itself)
      • use_isolated_script_api
• Cannot use variants

These conditions will be removed as more support is implemented.

To set tests for a builder in starlark, the builder must be setting the builder_spec argument. If that is the case, simply set the targets field of a builder to the name of a bundle or a list of bundle names to include. Supported target types create a bundle with the same name as the target, so single targets can be referenced wherever a bundle is expected.

ci.builder(
  name = "foo-builder",
  builder_spec = ...,
  # Configures the builder to build all
  targets = targets.bundle(
      additional_compile_targets = "all",
  ),
)

ci.builder(
  name = "bar-builder",
  builder_spec = ...,
  # Configures the builder to build all and run any tests and build any compile
  # targets specified by the public_build_scripts bundle
  targets = targets.bundle(
      additional_compile_targets = "all",
      targets = "public_build_scripts",
  ),
)

Fields that would be specified on the waterfall can instead be set on targets.builder_defaults, which will apply to all builders defined in the file.

targets.builder_defaults.set(
  mixins = [
    "chromium-tester-service-account",
    "linux-jammy",
  ],
)

This will cause json files containing the targets specs to be written out in the per-builder generated directory. Try builders that mirror other builders do not need to set the targets field (and it is an error to do so). The mirroring relationship indicates that it should use the targets configured for the builders that it mirrors, so mirroring try builders will have their own json files generated containing the logical union of the contents of the mirrored builders. This makes it more clear what builders are affected by modifications to a CI builder's targets.