diff --git a/daisy/README.md b/daisy/README.md index 1544baa35..18a21443f 100644 --- a/daisy/README.md +++ b/daisy/README.md @@ -36,12 +36,12 @@ Other use-case examples: * [CopyGCSObjects](#type-copygcsobjects) * [DeleteResources](#type-deleteresources) * [IncludeWorkflow](#type-includeworkflow) - * [RunTests](#type-runtests) * [SubWorkflow](#type-subworkflow) * [WaitForInstancesSignal](#type-waitforinstancessignal) * [Dependencies](#dependencies) * [Vars](#vars) * [Autovars](#autovars) + * [Testing](#testing) * [Glossary of Terms](#glossary-of-terms) * [GCE](#glossary-gce) * [GCP](#glossary-gcp) @@ -452,9 +452,6 @@ This IncludeWorkflow step example uses a local workflow file and passes a var, } ``` -#### Type: RunTests -Not implemented yet. - #### Type: SubWorkflow Runs a Daisy workflow as a step. The subworkflow will have some fields overwritten. For example, the subworkflow may specify a GCP Project "foo", @@ -623,6 +620,26 @@ out of convenience. Here is the exhaustive list of autovars: | OUTSPATH | Equivalent to ${SCRATCHPATH}/outs. | | USERNAME | Username of the user running the workflow. | +## Testing +Infrastructure has been set up to perform presubmit testing on PRs and +periodic continuous integration tests against HEAD. + +Presubmit checks unit tests, `golint`, `go fmt`, and `go vet` against PRs +with changes to Daisy source code. Unit test coverage is reported to +codecov.io, which posts coverage reports on the PR. Presubmit tests are +gated by repo owners. Repo owners have the following commands available on +a PR: +* `/go test`: runs unit tests and reports coverage. +* `/golint`: runs `golint`. +* `/go fmt`: runs `go fmt`. +* `/go vet`: runs `go vet`. +* `/ok-to-test`: gives Prow the go-ahead to run the entire suite automatically. +* `/retest`: reruns failed tests, only available after Prow reports failures. + +Periodic tests run every 6 hours. Currently, periodic tests include the e2e +tests here: [../daisy_workflows/e2e_tests](../daisy_workflows/e2e_tests). You +can see the test results for the e2e tests in testgrid: [https://k8s-testgrid.appspot.com/google-gce-compute-image-tools#daisy-e2e]. + ## Glossary of Terms Definitions: * GCE: Google Compute Engine diff --git a/daisy/prowjob/run/ci/__main__.py b/daisy/prowjob/run/ci/__main__.py index 9f7548a3c..c9eb48bfb 100644 --- a/daisy/prowjob/run/ci/__main__.py +++ b/daisy/prowjob/run/ci/__main__.py @@ -65,7 +65,7 @@ def build_subsuites(wfs): continue suite = match.group('suite') test_num = match.group('test_num') - test_num = int(test_num) if test_num else 0 + test_num = int(test_num) if test_num else -1 suites[suite] = suites.get(suite, []) + [(test_num, wf)] for suite in suites: diff --git a/daisy_workflows/e2e_tests/README.md b/daisy_workflows/e2e_tests/README.md new file mode 100644 index 000000000..97ccaa2c6 --- /dev/null +++ b/daisy_workflows/e2e_tests/README.md @@ -0,0 +1,31 @@ +# Daisy E2E Tests +This directory contains end to end tests for Daisy. These tests are run +periodically against HEAD. + +* Files matching `*.wf.json` are individual test cases. + * If you have a sub wf that you do not want to run as a test case, you need + to use a different file extension, e.g. `thing.subwf.json`, or put it in a + subdirectory, e.g. `subdir/thing.wf.json`. +* Test cases run in parallel except in cases of ordered test cases. +* Ordered test cases: + * Share a prefix. + * Use non-negative numeric suffixes (before the `.wf.json`). If there + is a test case without a suffix, it runs first. + * e.g. `foo.wf.json`, `foo0.wf.json`, `foo1.wf.json`, and `foo014.wf.json`: + `foo.wf.json` will run, then `foo0.wf.json`, then `foo1.wf.json`, then + finally `foo014.wf.json`. + +## Test Environment Details +* Tests run in the GCP project `gce-daisy-test` and have permissions: + * GCE read/write + * GCS read on the `gce-daisy-test-resources` bucket. + * GCS read/write on the `gce-daisy-test-sandbox` bucket. +* Test logs are written to the `gce-daisy-test` GCS bucket for Gubenator and + testgrid to pick up. +* Defaults are provided for workflow `Project` and `Zone` fields. + * `Project` is `gce-daisy-test` + * `Zone` is variable. +* The following args are passed to the test workflows: + * `test-id`: The ID of this test run. Useful for sharing resources between + test cases (would probably need to be ordered test cases, since parallel) + test cases would cause race conditions).