Upgrade tarpaulin for new features

[clux]

Want to use a less ancient version of tarpaulin and get new features.

• Bump to latest (currently 0.27.3) from our pin at 0.18.5 from Nov 2021
• Find a better GH Action approach (see below for the issues with actions-rs/tarpaulin) - taike-e install via https://github.com/tyrone-wu/kube/pull/1/files
• Try the proc macro coverage so that we can get credit for tests in kube-derive (currently it looks bad but it is tested)
• Do an all-features build in coverage (to ensure numbers are the most accurate and the most tests run)
• [~] Try to include one or two examples if it can be done quickly (better to improve coverage with actual integration tests)

A lot of this can be attempted locally by simply running cargo install cargo-tarpaulin then running just coverage inside the root of this repo (with a local k3d / non-important kubernetes cluster at your context).

Some of this has been attempted before. #1143 most recently. But it's annoying because: a) changes have surprising results, b) iteration cycles are slow, and c) we never got around many of the problems.

Problems encountered

• Upgrade from 0.18.5 to two tested versions (0.22 and 0.25 locally) both drops 13% code coverage because it fails to include everything and it seems to fail to exclude tests (ignore_tests = true). This needs to be fixed / worked around.
• actions-rs/tarpaulin will be deprecated and is otherwise not well maintained (breaks after 0.22.0 after which their asset resolution fails => we can only bump up to 0.22.0 from Oct 2022 using this action)
• Tarpaulin's GHA recommendation is to run their image in docker which seems to clash with the k3d action (a no-go because we rely on integration tests being run with tarpaulin) so we probably need a bin install (there's probably a good example available in GH search)
• The runtime-macros lib that can give us macro-expansion coverage had maintenance issues and as a result has an unmergeable fork

It is possible that there is a lot more work to be done upstream before we need to really do anything here - in particular, see the master issue about Misses in coverage. So in the interest of having a place to resume from down the line, I am stashing knowledge here for a potentially ambitious future me, or anyone who wants to play with this.

It's a minor nice-to-have CI issue (tarpaulin and the actions are already great helpers!) and it's not really costing us anything real by lagging behind for another year or three. At least as long as the nodejs 12 deprecation issue on CI from the action becomes breaking.

[tyrone-wu]

Hi! I'm currently giving this a stab and here's some of the things that I've found so far. 🐱

It looks like ignore-tests is working properly now and is the default behavior. I upgraded the coverage workflow to the new tarpaulin and I'm also getting the 13% difference on my codecov. From a couple side-by-side comparisons with your codecov and mine, I only difference I see is my tests are ignored in the coverage statistic.

drops 13% code coverage because it fails to include everything and it seems to fail to exclude tests (ignore_tests = true).

Do you recall which parts were not included?

Edit: After looking through the tarpaulin's code some more, it looks like the ignore-tests arg doesn't do anything, so it can't be disabled. I originally wanted to disable it to see if I could reach your coverage %, but it doesn't seem possible with the new tarpaulin. :(

[clux]

Hey thanks! Quick check now with new tarpaulin and i am getting data that excludes the test data, and as a result our coverage drops:

Locally with just coverage with latest tarpaulin installed:

kube		0/0 (100%)
kube-client	1038 / 1850 (56.11%)
kube-core	745 / 1009 (73.84%)
kube-derive	18 / 159 (11.32%)
kube-runtime	595 / 1076 (55.30%)

compared to live

kube		175 / 192 (91.15%)
kube-client	1651 / 2550 (64.75%)
kube-core	1398 / 1670 (83.71%)
kube-derive	68 / 210 (32.38%)
kube-runtime	1455 / 1970 (73.86%)

scanned like 3-4 files and all were due to the live count including the tests and local does not.

[clux]

Ah, have found a way to make it work locally in an acceptable way by moving the ignore-tests flag from the config to the cli:

diff --git a/justfile b/justfile
index eba15879..a73aa98e 100644
--- a/justfile
+++ b/justfile
@@ -46,7 +46,7 @@ test-integration:
   cargo run -p kube-examples --example crd_api
 
 coverage:
-  cargo tarpaulin --out=Html --output-dir=.
+  cargo tarpaulin --out=Html --output-dir=. --include-tests
   {{open}} tarpaulin-report.html
 
 readme:
diff --git a/tarpaulin.toml b/tarpaulin.toml
index 30bf94c4..638c9385 100644
--- a/tarpaulin.toml
+++ b/tarpaulin.toml
@@ -7,16 +7,15 @@
 
 [one_pass_coverage]
 workspace = true
-features = "kube/derive kube/runtime kube/ws"
+features = "kube/derive kube/runtime kube/ws kube/jsonpatch kube/admission kube/gzip"
 color = "Always"
 ignored = true
 timeout = "600s"
 exclude = ["e2e"]
 # NB: proc macro code is not picked up by tarpaulin - so could maybe skip kube-derive completely
-excluded_files = ["kube-derive/tests"]
+exclude-files = ["kube-derive/tests"]
 # NB: skipping Doctests because they are slow to build and generally marked no_run
 run-types = ["Tests"]
-ignore_tests = true
 
 # We could potentially pass in examples here
 # but: they don't help in covering kube-derive, and they force a full recompile

(also added some features to test in this run, but all-features also seems to work fine - but perhaps it's too slow to compile for the extra percent coverage it gives as it's already the slowest job)

and we get this coverage:

kube	33 / 34 (97.06%)
kube-client	1238 / 2056 (60.21%) (+4.11%)
kube-core	1399 / 1670 (83.77%) (+9.94%)
kube-derive	68 / 210 (32.38%) (+21.06%)
kube-runtime	684 / 1194 (57.29%) (+2.08%)

so maybe that makes it easier. i otherwise like your new workflow setup with taiki-e/install-action! so thanks for looking into this. is it slower / faster?

EDIT: using the cli arg is not necessary!

-excluded_files = ["kube-derive/tests"]
+exclude-files = ["kube-derive/tests"]
+include-tests = true
 # NB: skipping Doctests because they are slow to build and generally marked no_run
 run-types = ["Tests"]
-ignore_tests = true

also works without cli flags!

[clux]

took some of your changes for a quick test to see what we get and the results are a bit strange (via #1460)
the coverage run produces -5% (which is not too bad all things considering), but it's just very strange what is and isn't covered.
link to files with changed coverage

..maybe it's just the consequence of going onto all-features 🤔

[tyrone-wu]

Originally, I was only using GHA's host runners to run the tests since I was encountering some issues setting up the k3d cluster locally. 😅 But I was able to get that resolved (I needed to add @server:* to the last --k3s-arg).

I looked through upstream and #1460 codecovs, and it looks like upstream covers #[tokio::test] while #1460 does not (upstream vs code-tweak), which is weird considering that include-tests = true and #[test] are both being picked up fine. This is might be a bug from tarpaulin? and could be were the 5% is missing.

I made a comment about what we're encountering here. I suppose we'll have to wait until the tarpaulin author sees it, but another option is ignoring all tests in the coverage completely (although this would make coverage look a bit 🫠 even though nothings changed internally in the tests).

For now, I'll try and play around with macro expansion.