[ENH] Replace prts metrics

[aryanpola]

Reference Issues/PRs

Addresses #2066

What does this implement/fix? Explain your changes.

Implementation of Precision, Recall and F1-score metrics.

PR checklist

For all contributions

• I've added myself to the list of contributors. Alternatively, you can use the @all-contributors bot to do this for you.
• The PR title starts with either [ENH], [MNT], [DOC], [BUG], [REF], [DEP] or [GOV] indicating whether the PR topic is related to enhancement, maintenance, documentation, bugs, refactoring, deprecation or governance.

[aeon-actions-bot]

Thank you for contributing to aeon

I have added the following labels to this PR based on the title: [ $\color{#FEF1BE}{\textsf{enhancement}}$ ].
I have added the following labels to this PR based on the changes made: [ $\color{#264F59}{\textsf{benchmarking}}$ ]. Feel free to change these if they do not properly represent the PR.

The Checks tab will show the status of our automated tests. You can click on individual test runs in the tab or "Details" in the panel below to see more information if there is a failure.

If our pre-commit code quality check fails, any trivial fixes will automatically be pushed to your PR unless it is a draft.

Don't hesitate to ask questions on the aeon Slack channel if you have any.

PR CI actions

These checkboxes will add labels to enable/disable CI functionality for this PR. This may not take effect immediately, and a new commit may be required to run the new configuration.

• Run pre-commit checks for all files
• Run mypy typecheck tests
• Run all pytest tests and configurations
• Run all notebook example tests
• Run numba-disabled codecov tests
• Stop automatic pre-commit fixes (always disabled for drafts)
• Disable numba cache loading
• Push an empty commit to re-run CI checks

[MatthewMiddlehurst]

Some of these functions look like they should be private/protected. In the tests please run both the new and current functions to ensure the output is the same (using pytest)

[aryanpola]

Changes made from the previous commit:

1. Added a function for flattening lists of lists into a list.
2. Removes overlapping functions.
3. Changed the metric calculations from averaging(w.r.t cardinality) to global(w.r.t overlapping positions) calculation of real and pred ranges.

[aryanpola]

@MatthewMiddlehurst Please let me know if the test cases are fine, I can separate them into different functions if necessary.

[TonyBagnall]

thanks for this, its really helpful. We will look again next week

[aryanpola]

Can you identify the relevant code for "range-based recall" and "range-based precision" in https://github.com/IntelLabs/TSAD-Evaluator?

Yes yes, it's just that I got carried away trying to implement classic metrics (obviously not what the issue stated).
I've updated the code with the necessary changes and added more test cases which should cover everything.

[aryanpola]

Also, value errors will rise when gamma = udf_gamma in precision which wouldn't affect the precision calculation.

[MatthewMiddlehurst]

Its fine to use bits of documentation from the other function when applicable. I'll leave the input type call to Sebastian but bit odd to change it IMO.

Again i would like the output of the current functions verified against the current ones or the current functions to be used in testing for now just to verify that the output is the same.

[aryanpola]

@MatthewMiddlehurst There is repetition of documentation in public functions so changing them to point towards other functions would be a bad idea.
Let me know if you want me to change them.

[MatthewMiddlehurst]

I'm not quite sure what you mean. In case there is confusion of the original issue, the end goal is to remove the other version of these functions eventually.

[aryanpola]

I'm not quite sure what you mean. In case there is confusion of the original issue, the end goal is to remove the other version of these functions eventually.

The implementation supports range metrics (ts_precision, ts_recall, ts_fscore) and removes the dependency on prts package.

[aryanpola]

Also, there are some changes to be made:

1. Separate parameters in ts_fscore -> alpha, bias to p_alpha, r_alpha, p_bias, r_bias.
2. Remove udf_gamma from precision. Currently, it only raises errors.

I'll update in some time.

[aryanpola]

@MatthewMiddlehurst @SebastianSchmidl @TonyBagnall changes from my side are done, please review when free and suggest changes if any :)

[MatthewMiddlehurst]

The implementation supports range metrics (ts_precision, ts_recall, ts_fscore) and removes the dependency on prts package.

I may just be missing something here. Why would there be repetition in documentation if you use documentation from the functions we are removing and replacing. The prts functions are still present and prts is still a dependency for the package. The replacement functions do not take the same input. I would disagree with the old ones being removed in this PR currently, given the ones you have implemented are different and unverified.

Could you please:

• Describe why the new implementations have deviated from the original wrappers in terms of input https://github.com/aeon-toolkit/aeon/blob/main/aeon/benchmarking/metrics/anomaly_detection/_binary.py
• Write tests comparing the old and new versions to ensure they have the same output.

[aryanpola]

I may just be missing something here. Why would there be repetition in documentation if you use documentation from the functions we are removing and replacing.

I meant for the functions precision and recall in the newer implementation, they have the same docstrings.

[aryanpola]

The already existing implementation (_binary.py) takes inputs as binary values (arrays). The one we are trying to replace with, takes inputs as list of tuples. So the changes would be to convert the list of tuples to binary arrays for compatibility with already existing functions?

Thanks for pointing out the conversion, haven't checked the input data type for _binary.py.

[MatthewMiddlehurst]

It would be fine to accept both and convert to either internally.

[aryanpola]

@MatthewMiddlehurst @SebastianSchmidl I've added a function to check if the inputs are binary, if yes, they get converted to ranges based tuples for further calulations. Also added the existing test case from test_ad_metrics.py.

This can now take range based or binary inputs.

[aryanpola]

A bit of help needed with the errors. All the tests pass locally.

[aryanpola]

@MatthewMiddlehurst Thanks for the help!

Anymore changes necessary?

[SebastianSchmidl]

Write tests comparing the old and new versions to ensure they have the same output. previous comment

This is still missing, right?

[aryanpola]

Write tests comparing the old and new versions to ensure they have the same output. previous comment

This is still missing, right?

The updated test compares the results of the previous version (_binary.py) and the new version (range_metrics).

Also, let me know if you want me to remove some test cases, these cover all the edge cases.