Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change needsdiagnosis model to consider moved milestone as needsdiagnosis = True #2805

Closed
ksy36 opened this issue Mar 10, 2022 · 0 comments
Closed

Comments

@ksy36
Copy link
Contributor

ksy36 commented Mar 10, 2022

We have observed that needsdiagnosis model classifies certain issues that potentially need diagnosis as needsdiagnosis = False. While this is expected, I think it might be getting worse, as the issues data is unbalanced and has much more data points for needsdiagnosis = False. We've started a discussion in mozilla/webcompat-team-okrs#256

It's worth mentioning that in the recent sample of 22 issues most of the issues that looked like false negatives didn't end up needing diagnosis (not reproducible, out of the scope of the project, etc.), so they're true negatives (prediction was correct for 21 of them). We'll continue tracking them to get a more representative sample.

As an example, issues that looked like they need diagnosis, but in the end, they didn't (prediction was correct):
webcompat/web-bugs#100746
webcompat/web-bugs#100676
webcompat/web-bugs#100687

Issues that are false negatives (prediction was incorrect):
webcompat/web-bugs#100495
webcompat/web-bugs#100645

I was thinking of including an additional set of issues that will contribute to needsdiagnosis = True pool. We have recently added a moved milestone. These issues often don't need diagnosis and are moved to bugzilla or elsewhere, but their content should be contributed to needsdiagnosis = True rather than false.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant