Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Restructure tests to reduce duplication and improve runtimes #680

Merged
merged 12 commits into from
Jan 24, 2024
Merged

Conversation

dzuelke
Copy link
Contributor

@dzuelke dzuelke commented Jan 24, 2024

tl;dr: fewer test lines, and tests now take about 5:30-6:30 minutes instead of 7:00-8:00 minutes.

Remove duplication mostly for SIGTERM related tests, and split some tests up into smaller/faster test files to improve parallel_tests' ability to balance test files around based on their logged runtimes.

Test file names are now more consistent (php_concurrency_shared.rb as the base, php-8.3_concurrency-nginx_spec.rb as an "instance" for PHP 8.3 and Nginx).

Sped up some tests, in particular the one for request_terminate_timeout, which is now ~25 seconds faster than before.

Runs will now automatically use ceil($numTestsForStack/2) (based on the runtime log) parallel processes to ensure that there are enough workers for the few really slow tests to run standalone, while at the same time not having faster tests pile up for longer than the slowest test.
This is better than a hard-coded process count, since it means

  1. we have fewer parallel processes (and thus less Heroku load) for heroku-22 since there are fewer tests (for fewer supported PHP versions), and
  2. we do not have to remember adjusting things downward when heroku-20 goes EOL (and its tests get removed), or
  3. adjust things upwards when a future PHP version is added (which again increases the number of tests).

GUS-W-14875944

@dzuelke dzuelke requested a review from a team as a code owner January 24, 2024 13:30
Base automatically changed from boot-scripts-test-option to main January 24, 2024 14:42
the name of the file has never made any sense

drop the absolute paths test, since it'd be caught by many others (CI etc)
No updates to that project since 2017; more official deprecation can be done next
undercores for name parts, hyphens for variants
ci_spec.rb is the slowest file by far, so this will help with overall test run times
we set a low manual limit for termination, and check via the new boot script config dumper that the defaults are 3s and 30s for log and term
for better balancing in parallel_tests
for better parallelization
it executes a few more tests, for PHP 7.4 and Composer 1
@dzuelke dzuelke merged commit cd7fbbb into main Jan 24, 2024
4 checks passed
@dzuelke dzuelke deleted the ci-speedup branch January 24, 2024 15:01
@dzuelke dzuelke restored the ci-speedup branch January 26, 2024 14:43
@heroku-linguist heroku-linguist bot mentioned this pull request Feb 9, 2024
@edmorley edmorley deleted the ci-speedup branch March 4, 2024 14:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants