Table in comment #8297
Table in comment #8297
Conversation
| ## Table | ||
| | |B2G Emulator Image Build|Mochitest Browser Chrome| | ||
| |---|:-:|:-:| | ||
| |b2g-emu-jb/debug|1| | |
There was a problem hiding this comment.
Should the number not be in the last cell (for debug)?
There was a problem hiding this comment.
No, I don't think so. It corresponds to the first job in the file https://github.com/mozilla/treeherder/blob/master/tests/sample_data/job_data.txt .
@Archaeopteryx What makes you think that?
Maybe the test data is not entirely consistent.
@jmaher Can you take a look, please ?
There was a problem hiding this comment.
I am looking into this today- the data in the file is years old, but should still be somewhat relevant. First look is we will want rows to be platform:config, and columns to be test_variants - but there is no clean way to define test variants; we can make progress now and refine in round 2.
There was a problem hiding this comment.
as commented in the specific code review- we need to:
- update the sample data with better task names, get rid of really old stuff, and add stuff with variants
- use code (provided sample) to find variants from the task name!
given those 2, I think the table fields will be more logical and concerns will go away! Thanks for working on this- while it seems "minor", I think it is a very big change for something that hasn't changed in 6+ years.
djangoliv
force-pushed
the
table-in-comment
branch
2 times, most recently
from
126a502
to
e10f317
Compare
| @@ -314,28 +334,41 @@ def get_bug_stats(self, startday, endday): | |||
|
|
|||
| for bug in bugs: | |||
| platform = bug["job__machine_platform__platform"] | |||
| test_variant = bug["job__signature__job_type_name"].rsplit("-", 1)[0] | |||
There was a problem hiding this comment.
this is a good shortcut, but will not work long term, here are a few cases to consider:
- we have job_type_name that is a set of tests run in parallel, so these end with numbers (i.e. mochitest-plain-1)
- some variants have multiple
-in them: https://searchfox.org/mozilla-central/search?q=suffix&path=variants.yml&case=false®exp=false - sometimes we don't have a variant at all;
of these, 1 and 2 need to be solved, we can hack around #3 as that is normal, but in the scope of parsing, it could be agreed as an "edge case".
as this is python code, we could copy it from another source (a phab request parsing task names to find variants, search for found_variants). When that code lands, I have some refactoring I want to do so eventually this will become an easier library to reuse.
- in the case of the example code, found_variants is a list, you can probably join them with a '+' or a '-' for the purposes of the table header.
we should update the sample test data in order to have more modern and useful task names.
The rest of the changes look good!
djangoliv
force-pushed
the
table-in-comment
branch
from
e10f317
to
ffa7d6e
Compare
djangoliv
force-pushed
the
table-in-comment
branch
from
f6db299
to
fd394cf
Compare
|
|
||
| with open("tests/intermittents_commenter/expected_comment_with_5_failures.text") as comment: | ||
| expected_comment = comment.read() | ||
| print(len(expected_comment)) |
There was a problem hiding this comment.
Debugging statements in this and next line
There was a problem hiding this comment.
Thanks, I also removed the corresponding lines in the previous test
|
please rebase from tip of tree; I had updated some sample_data a week ago; ideally you can get what you need for better test cases |
This is a first version for adding table in comments generated in bugzilla for intermittent tests.
For now, only failed tests appear in the table.
In a future MR, we will also be able to add successful tests.
We can also take advantage of this to improve the code of the get_bug_stats function.