From e787f952cc0cd4ca4aeac3249bdaeb56a753e7bc Mon Sep 17 00:00:00 2001 From: Scott Chamberlain Date: Thu, 23 Jan 2025 09:20:47 -0800 Subject: [PATCH 1/2] rearrange tests info in readme: add java specific instructions --- README.md | 56 +++++++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 48 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index 480011a..e41410d 100644 --- a/README.md +++ b/README.md @@ -27,26 +27,66 @@ This test suite is designed to: ## Contributing See our [CONTRIBUTING.md](.github/CONTRIBUTING.md) for exact details on how to contribute. -## Cromwell tests +## Tests -A suite of tests for running WDLs against Cromwell are located in `tests/cromwell-api`. Tests are run using Python. To run tests: +There are two sets of tests for running WDLs against Cromwell using Python: + +- `tests/cromwelljava`: Runs WDLs on Cromwell using a local copy of `cromwell-.jar` and `womtool-.jar`. See "Java" below for details. +- `tests/cromwellapi`: Runs WDLs on Cromwell running on FH clusters via the [PROOF API][proofapi]. See "API" below for details. + +What's the difference between the two sets of tests? The Java tests run against just the Java Cromwell code itself, whereas the API tests are a kind of integration test that incorporates all the infrastructure behind PROOF including the [PROOF API][proofapi], the Cromwell servers that PROOF manages, and the FH cluster infra that it all runs on. + +To run both sets of tests: - clone this repository locally and cd into it - if you don't have `uv` installed, [install it][uvinstall] -- create a virtual environment: `uv venv`. This should pick up the Python version specified in `.python-version`. If you do not have the version given in `.python-version` it will be installed by `uv`. +- create a virtual environment: `uv venv`. This should pick up the Python version specified in `.python-version`. If you do not have the version given in `.python-version` it will be installed by `uv` - activate the virtual environment: `source .venv/bin/activate` - check that you have the Python version in `.python-version`: `python --version` + +Info: +- [pytest][] +- [uv][] + +### Java + +These tests are run on GitHub Actions on each pull request via the `.github/workflows/cromwell-test-run.yml` and `.github/workflows/womtools-validate.yml` workflows. + +To run them locally: + +- You'll need env vars set for the paths for each of `cromwell-.jar` and `womtool-.jar`. For cromwell it's `CROMWELL_PATH` and for womtools it's `WOMTOOL_PATH`. You can either set the path in a dot file or export for the current shell session or prefix the pytest line below with the env key and value. + +```sh +CROMWELL_PATH=/path/to/cromwell-87.jar uv run pytest tests/cromwelljava/test-run.py --wdl-path=helloHostname --verbose -s + +WOMTOOL_PATH=/path/to/womtool-87.jar uv run pytest tests/cromwelljava/test-validate.py --wdl-path=helloHostname --verbose -s +``` + +Note that each call to `test-run.py` or `test-validate.py` runs one WDL only. + +Adding tests shouldn't require any changes to the files in `tests/cromwelljava` - though if you run into any issues let's talk about it. + +### API + +These tests are run on GitHub Actions on each push to main/dev and each pull request via the `.github/workflows/api-tests.yml` workflow. + +To run them locally: + +- get on the Cisco VPN if working remotely or be on an FH campus network - set a PROOF token for the test user used for these tests as `PROOF_API_TOKEN_DEV`. the value is in 1password ([Secrets and env vars in 1password](https://developer.1password.com/docs/cli/secrets-environment-variables/)). Talk to Sean or Scott if you don't have access. 1. Open 1password and find the API key. 2. Right-click the key and select "Copy Secret Reference." 3. Run: `export PROOF_API_TOKEN_DEV="op://secret/reference/copied/from/1PW"` -- run tests, for example: `op run -- uv run pytest tests/cromwell-api --verbose` +- run tests, for example: `op run -- uv run pytest tests/cromwellapi --verbose` To add tests: -- Tests must be in files beginning with `test-`, and be in the `tests/` dir. All tests against the Cromwell API should go into `tests/cromwell-api/` dir -- Ideally use the pytest fixture `cromwell_api` created in `tests/cromwell-api/conftest.py` to call the Cromwell API. If there's a method missing that you need add it to the `CromwellApi` class in `tests/cromwell-api/cromwell.py` -- Make sure the tests pass. -- Put changes to code on a branch, then open a PR +- Tests must be in files beginning with `test-`, and be in the `tests/cromwellapi/` dir +- Ideally use the pytest fixture `cromwell_api` created in `tests/cromwellapi/conftest.py` to call the Cromwell API. If there's a method missing that you need add it to the `CromwellApi` class in `tests/cromwellapi/cromwell.py` + +Note that some of these tests run through all WDLs in this dir, while others only run through one or some subset of WDLs. [uvinstall]: https://docs.astral.sh/uv/getting-started/installation/ +[proofapi]: https://github.com/FredHutch/proof-api +[pytest]: CROMWELL_PATH=/Users/schambe3/github/cromwell/cromwell-86.jar +[uv]: https://docs.astral.sh/uv/ From 1cf2c8389a223a3c12230db75e199415a26502c0 Mon Sep 17 00:00:00 2001 From: Scott Chamberlain Date: Fri, 24 Jan 2025 10:27:59 -0800 Subject: [PATCH 2/2] changes from sean's comments --- README.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index e41410d..08d5dfb 100644 --- a/README.md +++ b/README.md @@ -42,11 +42,12 @@ To run both sets of tests: - if you don't have `uv` installed, [install it][uvinstall] - create a virtual environment: `uv venv`. This should pick up the Python version specified in `.python-version`. If you do not have the version given in `.python-version` it will be installed by `uv` - activate the virtual environment: `source .venv/bin/activate` +- install dependencies: `uv sync` - check that you have the Python version in `.python-version`: `python --version` -Info: -- [pytest][] -- [uv][] +Familiarity with two tools will help you run and especially contribute to tests: +- [pytest][]: a Python test framework. when you run `uv sync` above you'll then have a cli binary `pytest` available +- [uv][]: a Python package and project manager; manages dependencies, runs tests, etc. ### Java @@ -88,5 +89,5 @@ Note that some of these tests run through all WDLs in this dir, while others onl [uvinstall]: https://docs.astral.sh/uv/getting-started/installation/ [proofapi]: https://github.com/FredHutch/proof-api -[pytest]: CROMWELL_PATH=/Users/schambe3/github/cromwell/cromwell-86.jar +[pytest]: https://docs.pytest.org/en/stable/ [uv]: https://docs.astral.sh/uv/