Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

When scores are all 0, rank is always set to 2 #623

Closed
maximusunc opened this issue May 24, 2024 · 4 comments
Closed

When scores are all 0, rank is always set to 2 #623

maximusunc opened this issue May 24, 2024 · 4 comments

Comments

@maximusunc
Copy link

What is the purpose of rank in the ARS results, and how is it calculated? I would expect them all to either be unique or to be 1?

@MarkDWilliams
Copy link
Collaborator

There are some odd parts in how tie-breaking "pushes down" ranks on tie. The behavior is all in scoring.py, but that code was written by Rosina's team and just ported over to the ARS. I'm open to changing how ties are handled.

@maximusunc
Copy link
Author

Is rank used anywhere? UI? I was kind of assuming it was just the index of the result, but it sounds more complicated. Anyways, this came about because of the automated tests reports, so I made an issue there to not use rank in those reports: https://github.com/NCATSTranslator/ARS_Test_Runner/issues/30

@MarkDWilliams
Copy link
Collaborator

MarkDWilliams commented May 24, 2024

I'd have to double check with UI about whether they're using rank, but yeah it's more complicated than an index. I believe it was intended for some tie breaking behavior with weighted mean, though it doesn't work if the tie is the whole set

@ShervinAbd92
Copy link
Collaborator

Gus just confirmed that UI team doesnt use the rank returned from ARS, they calculate rank on their end.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants