Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(ensure): improve handling of metric names with special chars #2180

Conversation

bernardobridge
Copy link
Contributor

@bernardobridge bernardobridge commented Sep 29, 2023

Why

Originally reported here: #2166 (comment), ensure is currently broken with most Playwright test assertions on web vital metrics.

The problem is more general though - any custom metric that includes special characters can hit this.

Approach

As it's hard to predict what patterns can fall into this (as it depends on how the package filtrex is handling it), rather than going with an approach of replacing specific patterns, we try to identify the metricNames in the expression and apply a hash to them, so that filtrex can run without issues. Otherwise we'd have to deal with finding all possible patterns that might have an issue with filtrex - this way we only have to concern ourselves with being able to correctly isolate metricNames.

Breaking Change

I am not sure if this is a behaviour we currently support or not, as all examples we have don't use it, but in order to use this approach, the code assumes that there will be spacing used between any operators used. This is consistent with what we've documented here: https://www.artillery.io/docs/reference/extensions/ensure.

I have been able to change the approach to be able to support spacing, so that the following is now possible:

(((vusers.created-vusers.completed)/ceil(vusers.created) * 100) + vusers.slept)<= 0

However, this comes with the following downside: all operators listed here are now reserved keywords for the purposes of asserting using conditions, which means that metric names should not include them if you want to use ensure. This potentially includes Scenario Name, if you're asserting on vusers.created_by_name.

This was the best trade-off I could find.

Note: The exception to the above is that keywords like and, not, random etc can all be used within metric names (as long as they are used within words, as they were specially handled - otherwise it would not be possible to have custom metrics with words that included those.

Testing

Since I implemented automated acceptance tests here back in August, these were actually helping me spot mistakes along the way 👍. That being said, I thought this logic was the perfect candidate for unit tests too, so I actually wrote the unit tests before part of the refactor (half-TDD), so it's now well covered by that too.

Hopefully between these two, we are making sure all cases are covered. Please let me know if I might have missed something 🙇‍♂️!

@bernardobridge bernardobridge requested review from hassy and a team September 29, 2023 17:06
);
if (check.strict) {
global.artillery.suggestedExitCode = 1;
if (check.result !== 1) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ignore this change, it's just prettier. Prettier was failing on this file before due to a linting config problem, so it hasn't been applied here for a while

@bernardobridge bernardobridge marked this pull request as ready for review October 3, 2023 08:55
@bernardobridge
Copy link
Contributor Author

Closing this PR in favour of #2284

@bernardobridge bernardobridge deleted the bernardobridge/art-1375-fix-ensure-plugin-handling-of-metrics-with-rogue-characters branch June 10, 2024 14:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant