Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Restructure CI tests #353

Merged
merged 4 commits into from
Sep 1, 2023
Merged

Restructure CI tests #353

merged 4 commits into from
Sep 1, 2023

Conversation

Ogeon
Copy link
Owner

@Ogeon Ogeon commented Sep 1, 2023

This splits the tests up a bit and makes some of them run faster by skipping re-running the same tests. Also adds some new checks.

  • Full tests only with MSRV, on Linux, Windows and MacOS. Feature tests skip documentation tests.
  • Miri still runs with most tests, skipping some individual tests that are too slow.
  • Quick checks with stable, beta and nightly on Linux.
  • #[no_std] test runs as a separate job.
  • Checks with clippy.
  • Track test coverage.
  • Add CodSpeed to see how it works. May revert later if it's not better than before.

@Ogeon Ogeon added the internal An internal change that will not be shown in release notes label Sep 1, 2023
@Ogeon Ogeon force-pushed the improve_ci branch 2 times, most recently from 4604766 to a8face0 Compare September 1, 2023 12:28
@github-actions
Copy link

github-actions bot commented Sep 1, 2023

Benchmark for 44b573f

Click to view benchmark
Test Base PR %
Cie family/lab to lch 3.8±0.09µs 3.7±0.13µs -2.63%
Cie family/lab to xyz 915.3±12.17ns 895.7±16.06ns -2.14%
Cie family/lch to lab 1343.2±52.39ns 1433.3±192.22ns +6.71%
Cie family/linsrgb to xyz 351.7±16.37ns 350.4±13.58ns -0.37%
Cie family/linsrgb to xyz - wide::f32x8 288.4±9.89ns 291.1±11.04ns +0.94%
Cie family/xyz to lab 9.4±0.30µs 9.5±0.32µs +1.06%
Cie family/xyz to yxy 748.3±22.69ns 778.4±21.56ns +4.02%
Cie family/yxy to xyz 733.6±11.61ns 718.5±11.71ns -2.06%
Matrix functions/matrix_inverse 15.7±0.07ns 14.8±0.48ns -5.73%
Matrix functions/multiply_3x3 11.6±0.17ns 10.5±0.30ns -9.48%
Matrix functions/multiply_rgb_to_xyz 9.8±0.27ns 9.2±0.33ns -6.12%
Matrix functions/multiply_xyz 6.8±0.12ns 6.6±0.20ns -2.94%
Matrix functions/multiply_xyz_to_rgb 7.0±0.15ns 6.6±0.21ns -5.71%
Matrix functions/rgb_to_xyz_matrix 31.2±0.21ns 30.5±0.54ns -2.24%
Rgb family/hsl to hsv 940.3±0.42ns 880.8±21.56ns -6.33%
Rgb family/hsl to linear hsl 9.0±0.05µs 8.6±0.20µs -4.44%
Rgb family/hsl to rgb 2.9±0.00µs 2.8±0.06µs -3.45%
Rgb family/hsv to hsl 1171.5±1.07ns 1099.6±22.05ns -6.14%
Rgb family/hsv to hwb 278.6±3.92ns 263.4±9.77ns -5.46%
Rgb family/hsv to linear hsv 8.3±0.02µs 8.0±0.39µs -3.61%
Rgb family/hsv to rgb 2.5±0.00µs 2.3±0.08µs -8.00%
Rgb family/hwb to hsv 894.6±1.28ns 833.4±26.07ns -6.84%
Rgb family/hwb to linear hwb 9.8±0.10µs 9.0±0.23µs -8.16%
Rgb family/linear hsl to hsl 9.0±0.04µs 8.5±0.17µs -5.56%
Rgb family/linear hsv to hsv 8.4±0.01µs 7.9±0.16µs -5.95%
Rgb family/linear hwb to hwb 9.8±0.11µs 9.0±0.25µs -8.16%
Rgb family/linsrgb to rgb 4.5±0.03µs 4.2±0.13µs -6.67%
Rgb family/linsrgb to rgb - wide::f32x8 6.0±0.02µs 5.7±0.51µs -5.00%
Rgb family/linsrgb_f32 to rgb_u8 1131.7±10.25ns 1106.1±26.13ns -2.26%
Rgb family/linsrgb_f64 to rgb_u8 1451.2±11.93ns 1429.4±76.24ns -1.50%
Rgb family/rgb to hsl 753.8±2.05ns 707.7±22.42ns -6.12%
Rgb family/rgb to hsl - wide::f32x8 571.4±1.23ns 536.1±14.68ns -6.18%
Rgb family/rgb to hsv 584.4±0.39ns 544.2±19.85ns -6.88%
Rgb family/rgb to hsv - wide::f32x8 518.0±0.90ns 472.2±15.10ns -8.84%
Rgb family/rgb to linsrgb 4.6±0.02µs 4.5±0.08µs -2.17%
Rgb family/rgb to linsrgb - wide::f32x8 3.8±0.02µs 3.7±0.06µs -2.63%
Rgb family/rgb_u8 to linsrgb_f32 374.8±4.21ns 337.1±13.09ns -10.06%
Rgb family/rgb_u8 to linsrgb_f64 376.4±2.95ns 357.3±12.75ns -5.07%
Rgb family/xyz to linsrgb 445.2±0.44ns 433.0±8.74ns -2.74%
Rgb family/xyz to linsrgb - wide::f32x8 298.7±0.28ns 293.8±4.13ns -1.64%

@github-actions
Copy link

github-actions bot commented Sep 1, 2023

Benchmark for b89e8a4

Click to view benchmark
Test Base PR %
Cie family/lab to lch 3.1±0.00µs 3.1±0.00µs 0.00%
Cie family/lab to xyz 774.2±0.41ns 772.6±0.29ns -0.21%
Cie family/lch to lab 1146.7±36.52ns 1155.2±0.64ns +0.74%
Cie family/linsrgb to xyz 318.0±5.00ns 315.2±6.21ns -0.88%
Cie family/linsrgb to xyz - wide::f32x8 315.0±0.22ns 314.3±0.38ns -0.22%
Cie family/xyz to lab 8.3±0.01µs 8.3±0.01µs 0.00%
Cie family/xyz to yxy 657.9±0.31ns 666.4±0.57ns +1.29%
Cie family/yxy to xyz 611.7±1.72ns 611.9±2.06ns +0.03%
Matrix functions/matrix_inverse 14.6±0.02ns 14.6±0.02ns 0.00%
Matrix functions/multiply_3x3 10.4±0.00ns 10.4±0.00ns 0.00%
Matrix functions/multiply_rgb_to_xyz 9.6±0.00ns 9.6±0.01ns 0.00%
Matrix functions/multiply_xyz 6.4±0.00ns 6.4±0.00ns 0.00%
Matrix functions/multiply_xyz_to_rgb 6.4±0.00ns 6.4±0.00ns 0.00%
Matrix functions/rgb_to_xyz_matrix 29.3±0.02ns 29.1±0.02ns -0.68%
Rgb family/hsl to hsv 690.7±0.58ns 684.0±0.19ns -0.97%
Rgb family/hsl to linear hsl 7.5±0.01µs 7.5±0.00µs 0.00%
Rgb family/hsl to rgb 2.1±0.00µs 2.1±0.00µs 0.00%
Rgb family/hsv to hsl 952.0±0.58ns 953.6±0.32ns +0.17%
Rgb family/hsv to hwb 227.7±0.08ns 223.1±0.11ns -2.02%
Rgb family/hsv to linear hsv 7.0±0.00µs 7.0±0.00µs 0.00%
Rgb family/hsv to rgb 2.0±0.00µs 2.0±0.00µs 0.00%
Rgb family/hwb to hsv 555.5±0.22ns 555.3±0.22ns -0.04%
Rgb family/hwb to linear hwb 8.2±0.00µs 8.2±0.00µs 0.00%
Rgb family/linear hsl to hsl 7.6±0.00µs 7.6±0.00µs 0.00%
Rgb family/linear hsv to hsv 7.1±0.00µs 7.1±0.00µs 0.00%
Rgb family/linear hwb to hwb 8.2±0.00µs 8.2±0.00µs 0.00%
Rgb family/linsrgb to rgb 3.8±0.01µs 3.8±0.01µs 0.00%
Rgb family/linsrgb to rgb - wide::f32x8 5.5±0.00µs 5.4±0.00µs -1.82%
Rgb family/linsrgb_f32 to rgb_u8 913.5±0.48ns 911.5±0.37ns -0.22%
Rgb family/linsrgb_f64 to rgb_u8 1161.4±1.13ns 1161.0±0.88ns -0.03%
Rgb family/rgb to hsl 654.0±0.43ns 656.5±0.38ns +0.38%
Rgb family/rgb to hsl - wide::f32x8 487.8±0.53ns 487.7±0.45ns -0.02%
Rgb family/rgb to hsv 477.3±0.22ns 480.1±0.22ns +0.59%
Rgb family/rgb to hsv - wide::f32x8 428.7±0.93ns 429.3±0.25ns +0.14%
Rgb family/rgb to linsrgb 3.8±0.01µs 3.8±0.00µs 0.00%
Rgb family/rgb to linsrgb - wide::f32x8 3.6±0.00µs 3.6±0.01µs 0.00%
Rgb family/rgb_u8 to linsrgb_f32 284.8±0.16ns 288.2±0.15ns +1.19%
Rgb family/rgb_u8 to linsrgb_f64 285.5±0.24ns 285.5±0.12ns 0.00%
Rgb family/xyz to linsrgb 328.2±0.15ns 328.4±2.98ns +0.06%
Rgb family/xyz to linsrgb - wide::f32x8 307.1±0.37ns 307.1±0.24ns 0.00%

@Ogeon
Copy link
Owner Author

Ogeon commented Sep 1, 2023

Looks like the redundant clone lint is more lenient in later versions. We may want to disable it temporarily if it becomes annoying to work with. Hopefully only a problem now when nothing was checked.

@github-actions
Copy link

github-actions bot commented Sep 1, 2023

Benchmark for 026490f

Click to view benchmark
Test Base PR %
Cie family/lab to lch 3.7±0.17µs 3.5±0.17µs -5.41%
Cie family/lab to xyz 854.4±48.15ns 788.8±52.53ns -7.68%
Cie family/lch to lab 1683.4±92.65ns 1668.8±72.01ns -0.87%
Cie family/linsrgb to xyz 368.0±14.66ns 374.1±14.46ns +1.66%
Cie family/linsrgb to xyz - wide::f32x8 306.7±14.03ns 286.2±15.87ns -6.68%
Cie family/xyz to lab 9.3±0.39µs 10.1±0.59µs +8.60%
Cie family/xyz to yxy 829.1±66.07ns 761.5±61.16ns -8.15%
Cie family/yxy to xyz 704.7±35.22ns 690.4±55.26ns -2.03%
Matrix functions/matrix_inverse 14.2±1.23ns 13.5±0.72ns -4.93%
Matrix functions/multiply_3x3 10.2±0.91ns 10.6±0.46ns +3.92%
Matrix functions/multiply_rgb_to_xyz 9.0±0.48ns 9.6±0.68ns +6.67%
Matrix functions/multiply_xyz 6.0±0.26ns 5.9±0.46ns -1.67%
Matrix functions/multiply_xyz_to_rgb 6.0±0.45ns 6.0±0.35ns 0.00%
Matrix functions/rgb_to_xyz_matrix 26.4±1.31ns 25.9±1.70ns -1.89%
Rgb family/hsl to hsv 764.6±55.23ns 848.4±46.41ns +10.96%
Rgb family/hsl to linear hsl 8.5±0.61µs 8.6±0.51µs +1.18%
Rgb family/hsl to rgb 2.7±0.17µs 2.6±0.17µs -3.70%
Rgb family/hsv to hsl 1138.8±92.91ns 1181.1±56.44ns +3.71%
Rgb family/hsv to hwb 236.7±16.51ns 241.8±17.18ns +2.15%
Rgb family/hsv to linear hsv 8.4±0.53µs 8.2±0.73µs -2.38%
Rgb family/hsv to rgb 2.5±0.87µs 2.3±0.15µs -8.00%
Rgb family/hwb to hsv 675.6±43.88ns 687.4±29.69ns +1.75%
Rgb family/hwb to linear hwb 10.0±0.66µs 9.7±0.48µs -3.00%
Rgb family/linear hsl to hsl 8.4±0.39µs 9.3±0.30µs +10.71%
Rgb family/linear hsv to hsv 7.5±0.50µs 8.6±0.50µs +14.67%
Rgb family/linear hwb to hwb 9.4±0.63µs 9.6±0.57µs +2.13%
Rgb family/linsrgb to rgb 4.2±0.24µs 4.6±0.22µs +9.52%
Rgb family/linsrgb to rgb - wide::f32x8 5.5±0.35µs 5.8±0.29µs +5.45%
Rgb family/linsrgb_f32 to rgb_u8 1056.0±82.00ns 1088.8±40.08ns +3.11%
Rgb family/linsrgb_f64 to rgb_u8 1261.6±79.61ns 1384.4±70.15ns +9.73%
Rgb family/rgb to hsl 857.4±36.67ns 820.0±57.87ns -4.36%
Rgb family/rgb to hsl - wide::f32x8 589.4±67.84ns 633.7±26.77ns +7.52%
Rgb family/rgb to hsv 560.1±32.29ns 655.1±23.37ns +16.96%
Rgb family/rgb to hsv - wide::f32x8 577.5±38.64ns 582.4±34.67ns +0.85%
Rgb family/rgb to linsrgb 4.7±0.37µs 4.5±0.28µs -4.26%
Rgb family/rgb to linsrgb - wide::f32x8 3.8±0.23µs 3.1±0.26µs -18.42%
Rgb family/rgb_u8 to linsrgb_f32 334.6±14.79ns 334.1±15.32ns -0.15%
Rgb family/rgb_u8 to linsrgb_f64 298.1±25.75ns 330.5±16.43ns +10.87%
Rgb family/xyz to linsrgb 426.1±21.47ns 401.1±20.80ns -5.87%
Rgb family/xyz to linsrgb - wide::f32x8 308.0±17.29ns 284.5±14.66ns -7.63%

@codecov-commenter
Copy link

codecov-commenter commented Sep 1, 2023

Codecov Report

❗ No coverage uploaded for pull request base (master@fc6c530). Click here to learn what that means.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master     #353   +/-   ##
=========================================
  Coverage          ?   78.22%           
=========================================
  Files             ?      115           
  Lines             ?    19751           
  Branches          ?    19751           
=========================================
  Hits              ?    15451           
  Misses            ?     4074           
  Partials          ?      226           

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@github-actions
Copy link

github-actions bot commented Sep 1, 2023

Benchmark for c1efd84

Click to view benchmark
Test Base PR %
Cie family/lab to lch 3.3±0.00µs 3.4±0.00µs +3.03%
Cie family/lab to xyz 770.2±0.94ns 771.9±1.30ns +0.22%
Cie family/lch to lab 1203.3±1.21ns 1203.3±1.37ns 0.00%
Cie family/linsrgb to xyz 308.5±0.23ns 312.9±0.71ns +1.43%
Cie family/linsrgb to xyz - wide::f32x8 250.4±0.75ns 251.0±0.51ns +0.24%
Cie family/xyz to lab 8.6±0.01µs 8.6±0.04µs 0.00%
Cie family/xyz to yxy 677.5±0.40ns 670.9±0.19ns -0.97%
Cie family/yxy to xyz 622.1±0.55ns 632.2±0.82ns +1.62%
Matrix functions/matrix_inverse 12.9±0.01ns 13.1±0.00ns +1.55%
Matrix functions/multiply_3x3 9.6±0.02ns 9.8±0.01ns +2.08%
Matrix functions/multiply_rgb_to_xyz 8.4±0.02ns 8.4±0.02ns 0.00%
Matrix functions/multiply_xyz 6.0±0.03ns 6.0±0.03ns 0.00%
Matrix functions/multiply_xyz_to_rgb 6.0±0.04ns 6.0±0.03ns 0.00%
Matrix functions/rgb_to_xyz_matrix 26.1±0.01ns 26.0±0.02ns -0.38%
Rgb family/hsl to hsv 783.8±0.40ns 782.8±0.33ns -0.13%
Rgb family/hsl to linear hsl 7.5±0.01µs 7.5±0.02µs 0.00%
Rgb family/hsl to rgb 2.4±0.00µs 2.4±0.00µs 0.00%
Rgb family/hsv to hsl 969.8±0.67ns 974.4±0.53ns +0.47%
Rgb family/hsv to hwb 235.8±2.39ns 231.4±2.35ns -1.87%
Rgb family/hsv to linear hsv 7.0±0.01µs 7.0±0.01µs 0.00%
Rgb family/hsv to rgb 2.1±0.00µs 2.1±0.00µs 0.00%
Rgb family/hwb to hsv 745.1±1.30ns 744.5±1.66ns -0.08%
Rgb family/hwb to linear hwb 8.3±0.01µs 8.3±0.01µs 0.00%
Rgb family/linear hsl to hsl 7.5±0.01µs 7.5±0.00µs 0.00%
Rgb family/linear hsv to hsv 7.0±0.00µs 7.0±0.00µs 0.00%
Rgb family/linear hwb to hwb 8.2±0.00µs 8.2±0.01µs 0.00%
Rgb family/linsrgb to rgb 3.7±0.00µs 3.8±0.00µs +2.70%
Rgb family/linsrgb to rgb - wide::f32x8 5.0±0.00µs 5.0±0.00µs 0.00%
Rgb family/linsrgb_f32 to rgb_u8 947.5±0.54ns 946.2±0.40ns -0.14%
Rgb family/linsrgb_f64 to rgb_u8 1214.1±0.73ns 1214.7±0.63ns +0.05%
Rgb family/rgb to hsl 623.3±0.41ns 633.9±0.36ns +1.70%
Rgb family/rgb to hsl - wide::f32x8 455.6±0.21ns 476.5±0.39ns +4.59%
Rgb family/rgb to hsv 482.8±0.29ns 492.1±0.24ns +1.93%
Rgb family/rgb to hsv - wide::f32x8 415.7±0.24ns 431.6±0.16ns +3.82%
Rgb family/rgb to linsrgb 3.8±0.00µs 3.9±0.00µs +2.63%
Rgb family/rgb to linsrgb - wide::f32x8 3.1±0.00µs 3.2±0.00µs +3.23%
Rgb family/rgb_u8 to linsrgb_f32 286.2±1.07ns 286.2±1.03ns 0.00%
Rgb family/rgb_u8 to linsrgb_f64 293.0±1.30ns 291.9±1.36ns -0.38%
Rgb family/xyz to linsrgb 371.9±0.16ns 370.9±1.39ns -0.27%
Rgb family/xyz to linsrgb - wide::f32x8 248.8±0.18ns 248.5±0.34ns -0.12%

@Ogeon Ogeon force-pushed the improve_ci branch 2 times, most recently from 9983ef0 to fa54ba7 Compare September 1, 2023 14:26
@codspeed-hq
Copy link

codspeed-hq bot commented Sep 1, 2023

CodSpeed Performance Report

Congrats! CodSpeed is installed 🎉

🆕 0 new benchmarks were detected.

You will start to see performance impacts in the reports once the benchmarks are run from your default branch.

Detected benchmarks

@Ogeon
Copy link
Owner Author

Ogeon commented Sep 1, 2023

bors r+

@bors
Copy link
Contributor

bors bot commented Sep 1, 2023

Build succeeded!

The publicly hosted instance of bors-ng is deprecated and will go away soon.

If you want to self-host your own instance, instructions are here.
For more help, visit the forum.

If you want to switch to GitHub's built-in merge queue, visit their help page.

@bors bors bot merged commit 3a9fc0e into master Sep 1, 2023
14 checks passed
@bors bors bot deleted the improve_ci branch September 1, 2023 15:08
bors bot added a commit that referenced this pull request Sep 1, 2023
354: Actually interface with codspeed r=Ogeon a=Ogeon

Forgot this part in #353.

Co-authored-by: Erik Hedvall <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
internal An internal change that will not be shown in release notes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants