Support test_io_data for MultiDataEval and eval util #84
Conversation
MischaPanch
force-pushed
the
infinimol-dev
branch
from
a7d1005
to
41e2082
Compare
src/sensai/evaluation/eval_util.py
Outdated
| """ | ||
| self.io_data_dict = io_data_dict | ||
| self.test_io_data_dict = test_io_data_dict or {} |
There was a problem hiding this comment.
To avoid errors, we should check for a matching key set here.
There was a problem hiding this comment.
The docstring should also be adjusted to state this requirement. I don't think it's a valid use case for the user to want to use a user-defined test set for only some of the datasets (all or nothing, I'd say). If the users wants None, it has to be explicit rather than implicit.
There was a problem hiding this comment.
The docstring of io_data_dict should also state that for the case where test_io_data_dict is given, it specifies the training set to use and otherwise the full dataset, which will be split, depending on the mode, according to the rules defined in evaluator_params or cross_validator_params. Since the latter case is (likely) prevalent, we should mention it first.
Also, we should state that test_io_data_dict can only be used for single-split evaluations, not for cross-validation. If someone calls the evaluation with cross-validation enabled while test_io_data_dict is not None, we should raise an error.
|
The pipeline was successful, just the post-job cleanup failed for some reason. Can be merged |
Closes: #83
This should be uncontroversial