GitLab CI: speed up execution

[mr-c]

The entire pipeline finishes in 45 fewer minutes.
Compare job times with this PR to job times without this PR.

• gitlab-ci: add collapsable sections (with recorded durations)
• gitlab-ci: increase test threads to 4; standalone tests only need one test thread; use worksteal more often; upgrade cwltest, run conformance tests in parallel; enable caching of pip & mypy
• wdl tests: use pytest fixtures to support parallel execution

Changelog Entry

To be copied to the draft changelog by merger:

• PR submitter writes their recommendation for a changelog entry here

Reviewer Checklist

• Make sure it is coming from issues/XXXX-fix-the-thing in the Toil repo, or from an external repo.
  • If it is coming from an external repo, make sure to pull it in for CI with:

    contrib/admin/test-pr otheruser theirbranchname issues/XXXX-fix-the-thing
    

  • If there is no associated issue, create one.
• Read through the code changes. Make sure that it doesn't have:
  • Addition of trailing whitespace.
  • New variable or member names in camelCase that want to be in snake_case.
  • New functions without type hints.
  • New functions or classes without informative docstrings.
  • Changes to semantics not reflected in the relevant docstrings.
  • New or changed command line options for Toil workflows that are not reflected in docs/running/{cliOptions,cwl,wdl}.rst
  • New features without tests.
• Comment on the lines of code where problems exist with a review comment. You can shift-click the line numbers in the diff to select multiple lines.
• Finish the review with an overall description of your opinion.

Merger Checklist

• Make sure the PR passed tests, including the Gitlab tests, for the most recent commit in its branch.
• Make sure the PR has been reviewed. If not, review it. If it has been reviewed and any requested changes seem to have been addressed, proceed.
• Merge with the Github "Squash and merge" feature.
  • If there are multiple authors' commits, add Co-authored-by to give credit to all contributing authors.
• Copy its recommended changelog entry to the Draft Changelog.
• Append the issue number in parentheses to the changelog entry.

[mr-c]

The CI passes, this is ready for review and merging; thank you!

Runs almost one hour faster (!) than the previous pipeline:

https://ucsc-ci.com/databiosphere/toil/-/pipelines/9396 (151 minutes) vs
https://ucsc-ci.com/databiosphere/toil/-/pipelines/9389 (203 minutes)

[adamnovak]

I like the Gitlab changes and the parallelization.

I'm not wild about the implicit connections you get from fixtures magically supplying an argument (e.g. tmp_path when I went to grep for it is not defined in any of the files here, but also does not appear in any import statements, so you sort of have to just know it comes from https://docs.pytest.org/en/6.2.x/tmpdir.html). But I see why fixtures will be better for bringing in things for tests than the single setUp/tearDown that need to be the same for all the related tests in a class. So I can buy into moving to them.

I was worried dropping assertEqual would lose us the feature where we can actually see the mismatching values, but it turns out pytest will rewrite the bytecode in your assert statements (!) to make that work. (Grumble grumble what happened to "explicit is better than implicit" grumble grumble.)

I'm less keen on the move away from ToilTest, and the idea of type-checking the test files.

I liked the principle that all our test classes extend ToilTest. It still says on there:

toil/src/toil/test/__init__.py

Lines 57 to 59 in 2de6eea

	A common base class for Toil tests.
	Please have every test case directly or indirectly inherit this one.

It also provides the implementation for TOIL_TEST_TEMP which is documented as letting you control the temp dir for Toil's tests.

If we no longer want ToilTest to be the One True Base Class for test classes, we should change that docstring that says it is, and officially drop or otherwise replace features like TOIL_TEST_TEMP which it provides. (We probably could just drop that one; I'm not sure it's really used by anyone.)

On the type-checking, it can be informative to have the type hints in the test files, but the tradeoff between developer time and ability to catch lurking mistakes is very different in test code than in the code under test. It's less clear to me that the work of maintaining a full set of type hints in test code is worth it, since it's about as much fighting MyPy as for the code under test, but can never catch an error that would happen in a real run. (I guess it can save you test runs that will definitely fail because of type errors?) If there was a way for MyPy to look at the test code and fail it when it can show there will be a type error, but not when it's just not sure there won't be, that would probably be the appropriate level of static analysis for tests. But I don't think that's a thing we can put in the MyPy ignore file. So maybe since you've already taken the time to type hint these test files, we should just take them and hope maintaining the hints won't be too much work?

So overall I think the changes needed are:

1. Drop or fix TOIL_TEST_TEMP.
2. Demote ToilTest from being the One True Baseclass for tests, and either deprecate it in favor of not having a base class for tests, or add a new base class for Pytest-style tests to hold whatever remaining shared code we need.
3. Maybe a change to the contributing docs to state that we actually do want to move the project from Unittest test cases to Pytest ones, and that if you write a new test it should look like such-and-such?

[adamnovak]

This is one command, not a series of commands, right?