You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We have observed that needsdiagnosis model classifies certain issues that potentially need diagnosis as needsdiagnosis = False. While this is expected, I think it might be getting worse, as the issues data is unbalanced and has much more data points for needsdiagnosis = False. We've started a discussion in mozilla/webcompat-team-okrs#256
It's worth mentioning that in the recent sample of 22 issues most of the issues that looked like false negatives didn't end up needing diagnosis (not reproducible, out of the scope of the project, etc.), so they're true negatives (prediction was correct for 21 of them). We'll continue tracking them to get a more representative sample.
I was thinking of including an additional set of issues that will contribute to needsdiagnosis = True pool. We have recently added a movedmilestone. These issues often don't need diagnosis and are moved to bugzilla or elsewhere, but their content should be contributed to needsdiagnosis = True rather than false.
The text was updated successfully, but these errors were encountered:
ksy36
added a commit
to ksy36/bugbug
that referenced
this issue
Mar 10, 2022
We have observed that needsdiagnosis model classifies certain issues that potentially need diagnosis as
needsdiagnosis = False
. While this is expected, I think it might be getting worse, as the issues data is unbalanced and has much more data points forneedsdiagnosis = False
. We've started a discussion in mozilla/webcompat-team-okrs#256It's worth mentioning that in the recent sample of 22 issues most of the issues that looked like false negatives didn't end up needing diagnosis (not reproducible, out of the scope of the project, etc.), so they're true negatives (prediction was correct for 21 of them). We'll continue tracking them to get a more representative sample.
As an example, issues that looked like they need diagnosis, but in the end, they didn't (prediction was correct):
webcompat/web-bugs#100746
webcompat/web-bugs#100676
webcompat/web-bugs#100687
Issues that are false negatives (prediction was incorrect):
webcompat/web-bugs#100495
webcompat/web-bugs#100645
I was thinking of including an additional set of issues that will contribute to
needsdiagnosis = True
pool. We have recently added amoved
milestone. These issues often don't need diagnosis and are moved to bugzilla or elsewhere, but their content should be contributed toneedsdiagnosis = True
rather than false.The text was updated successfully, but these errors were encountered: