Skip to content

Latest commit

 

History

History
198 lines (160 loc) · 9.72 KB

README.md

File metadata and controls

198 lines (160 loc) · 9.72 KB
Raw

Render Templates

Overview

The Render Templates is a tool that reads the configuration from a config.yaml file and data files to generate output files, such as Prow component jobs. While the config.yaml file can hold configuration for an output file, you can place such data within the data files that hold configuration for related output files. Having separate files with grouped data is cleaner and easier to maintain than one huge config file.

The config.yaml file and the data files specify the following for the Render Templates:

  • Templates it must use to generate the output files
  • The name and location of the output files
  • Values it must use to generate the output files

Data for templates

The Render Templates passes data in the $.Values and $.Global variables to the templates to generate files. The values of these variables are created from the config.yaml file and data files. The Global variable holds data from the global key in config.yaml. The Render Templates generates the values of the Values variable from scratch for each Prow job. Values variable is generated by merging Config Sets. Each Config Set is a map defined in one of three possible places:

  • Global ConfigSets defined in the globalSets key in the config.yaml file:

    globalSets:
  image_bootstrap:
    image: "eu.gcr.io/kyma-project/test-infra/bootstrap:v20200831-e46c648b"

    Config Sets defined in globalSets hold data used to generate multiple files. A good example of such usage is the image_bootstrap global Config Set, which defines a bootstrap image to use in Prow jobs.

  • Local ConfigSets defined under the localSets parameter for each to key in the config.yaml file or in data files in the templates/data directory:

    templates:
    render:
      - to: ../prow/jobs/test-infra/buildpack.yaml
        localSets:
          default:
            skip_report: "false"
            max_concurrency: "10"
            branches:
            - "^main$"
          presubmit:
            type_presubmit: "true"
            labels:
              preset-build-pr: "true"
          postsubmit:
            type_postsubmit: "true"
            cluster: "trusted-workload"

    Config Sets defined in localSets have a scope limited to the generated file in which they are defined. Use localSets to hold data that is common within the generated file.

  • One-job ConfigSets defined in the jobConfig key:

    jobConfigs:
  - repoName: "kyma-project/test-infra"
    jobs:
    - jobConfig:
        name: "pre-test-infra-bootstrap"
        run_if_changed: "^prow/images/bootstrap/"
        args:
        - "/home/prow/go/src/github.com/kyma-project/test-infra/prow/images/bootstrap"

    Config Sets defined in jobConfig set data for one job. Use such Config Sets to keep values specific for one job only.

    Every job under the inheritedConfigs key specifies which Config Sets are inherited. This key holds a list of Config Sets names from globalSets and localSets.

    jobConfigs:
  - repoName: "kyma-project/test-infra"
    jobs:
      - jobConfig:
          name: "pre-test-infra-bootstrap"
          run_if_changed: "^prow/images/bootstrap/"
        inheritedConfigs:
          global:
            - "image_bootstrap"
          local:
            - "default"
            - "presubmit"

    Component job defined in jobConfig can be used to generate multiple job definitions for a single component. It is defined by having a path value, and by not having a name value. This type of config holds two additional lists of configSets named preConfigs and postConfigs, which hold lists of global and local Config Sets used for presubmit and postsubmit jobs.

    localSets:
  jobConfig_pre:
    labels:
      preset-build-pr: "true"
  jobConfig_post:
    labels:
      preset-build-main: "true"
jobConfigs:
  - repoName: "github.com/kyma-project/kyma"
    jobs:
      - jobConfig:
          path: components/application-gateway
          args:
            - "/home/prow/go/src/github.com/kyma-project/kyma/components/application-gateway"
          run_if_changed: "^components/application-gateway/|^common/makefiles/"
          release_since: "1.7"
        inheritedConfigs:
          global:
            - "jobConfig_default"
            - "image_buildpack-golang"
            - "jobConfig_generic_component"
            - "jobConfig_generic_component_kyma"
            - "extra_refs_test-infra"
          preConfigs:
            global:
              - "jobConfig_presubmit"
            local:
              - "jobConfig_pre"
          postConfigs:
            global:
              - "jobConfig_postsubmit"
              - "disable_testgrid"
            local:
              - "jobConfig_post"

    The Render Templates tool can generate precommit and postcommit job definitions from a single jobConfig. The job defined in jobConfigPre and jobConfigPost generates precommit and postcommit job definitions for a single job. This type of config holds two additional lists of values named jobConfigPre and jobConfigPost, which hold values used for presubmit and postsubmit jobs, as well as two lists of Config Sets named preConfigs and postConfigs, which hold lists of global and local Config Sets used for presubmit and postsubmit jobs.

    jobConfigs:
  - repoName: kyma-project/control-plane
    jobs:
      - jobConfig:
          labels:
            preset-common: "true"
        jobConfigPre:
          name: pre-main-kcp-cli
          run_if_changed: "^tools/cli"
        jobConfigPost:
          name: post-main-kcp-cli
          labels:
            preset-build-artifacts-main: "true"
        inheritedConfigs:
          global:
            - "jobConfig_default"
          local:
            - "jobConfig_default_kcp"
          preConfigs:
            global:
              - "jobConfig_presubmit"
          postConfigs:
            global:
              - "jobConfig_postsubmit"
              - "disable_testgrid"

The Render Templates tool builds the Values variable in the following order:

First, Config Sets from globalSets are merged. If the job inherits the default Config Set from globalSets, it is merged first and all other Config Sets from globalSets are merged afterwards.

Secondly, the Render Templates merges Config Sets from localSets. Again, if the job inherits the default Config Set from localSets, it's merged first and then all the other Config Sets from localSets are merged.

Config Sets other than default are merged in any order during the globalSets and localSets phases.

Finally, Config Sets from jobConfig are merged as the last ones.

Existing keys in the Values variable are overwritten by values from the merged Config Sets.

Usage

To run this tool, use one of these commands:

go run development/tools/cmd/rendertemplates/main.go --data path/to/directory/with/data/files

or

make jobs-definitions

By default, the Render Templates downloads config.yaml and templates files from GitHub. You can specify paths to the config.yaml file and templates directory on the command line to override defaults.

To work with local files only, provide all paths in flags.

Example for test-infra repository:

go run development/tools/cmd/rendertemplates/main.go --config templates/config.yaml --templates templates/templates --data templates/data

Flags

This tool uses one flag:

Name Required Description
‑‑config No Path to the Render Templates configuration file.
‑‑data No Path to directory with data files. Mutual exclusive with --data-file argument
‑‑data-file No Path to the data file. Mutual exclusive with --data argument.
‑‑templates No Path to the directory with templates files.
‑‑gh-token No GitHub personal access token. Use it to get files from GitHub as an authenticated user. By default the Render Templates access GitHub as an anonymous user.
‑‑show-output-dir No Prints out the paths to data files and to the generated files.
‑‑append-slice No Append slices instead of overwriting them when merging.