fix(ensure): improve handling of metric names with special chars #2180
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why
Originally reported here: #2166 (comment),
ensure
is currently broken with most Playwright test assertions on web vital metrics.The problem is more general though - any custom metric that includes special characters can hit this.
Approach
As it's hard to predict what patterns can fall into this (as it depends on how the package
filtrex
is handling it), rather than going with an approach of replacing specific patterns, we try to identify themetricNames
in the expression and apply a hash to them, so thatfiltrex
can run without issues. Otherwise we'd have to deal with finding all possible patterns that might have an issue withfiltrex
- this way we only have to concern ourselves with being able to correctly isolate metricNames.Breaking Change
I am not sure if this is a behaviour we currently support or not, as all examples we have don't use it, but in order to use this approach, the code assumes that there will be spacing used between any operators used. This is consistent with what we've documented here: https://www.artillery.io/docs/reference/extensions/ensure.I have been able to change the approach to be able to support spacing, so that the following is now possible:
However, this comes with the following downside: all operators listed here are now reserved keywords for the purposes of asserting using
conditions
, which means that metric names should not include them if you want to useensure
. This potentially includes Scenario Name, if you're asserting onvusers.created_by_name
.This was the best trade-off I could find.
Note: The exception to the above is that keywords like and, not, random etc can all be used within metric names (as long as they are used within words, as they were specially handled - otherwise it would not be possible to have custom metrics with words that included those.
Testing
Since I implemented automated acceptance tests here back in August, these were actually helping me spot mistakes along the way 👍. That being said, I thought this logic was the perfect candidate for unit tests too, so I actually wrote the unit tests before part of the refactor (half-TDD), so it's now well covered by that too.
Hopefully between these two, we are making sure all cases are covered. Please let me know if I might have missed something 🙇♂️!