You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
If we compute a diff table by comparing the status column (the default mode for diff tables), only the status itself is taken into account, not the associated category. Usually, this is no problem because for a given tuple (task, status) the category is always the same. However, in some scenarios additional categories (correct-unconfirmed) are introduced into the result files (from outside of BenchExec) and then this no longer applies. But the diff table will omit the columns where the category is different but the status is the same.
Given that we provide some support for the category correct-unconfirmed and e.g. compute statistics for it, we should also take it into account for the diff table. Comparing (status, category) by default instead of just status is probably a good default.
The text was updated successfully, but these errors were encountered:
If we compute a diff table by comparing the status column (the default mode for diff tables), only the status itself is taken into account, not the associated category. Usually, this is no problem because for a given tuple
(task, status)
the category is always the same. However, in some scenarios additional categories (correct-unconfirmed
) are introduced into the result files (from outside of BenchExec) and then this no longer applies. But the diff table will omit the columns where the category is different but the status is the same.Given that we provide some support for the category
correct-unconfirmed
and e.g. compute statistics for it, we should also take it into account for the diff table. Comparing(status, category)
by default instead of juststatus
is probably a good default.The text was updated successfully, but these errors were encountered: