-
-
Notifications
You must be signed in to change notification settings - Fork 229
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Handle invalid neighbors in local search gracefully #773
Handle invalid neighbors in local search gracefully #773
Conversation
If the iterator returned by `ConfigSpace.util.get_one_exchange_neighborhood` reaches an invalid configuration, it raises a `ValueError` with the probability 5 %. The randomness is sampled using `np.random.random()`, so it is non-deterministic. See ConfigSpace/util.pyx:218. This commit ensures that the exception is caught and handled gracefully.
In local search, only advance to a better-scoring neighbor if that neighbor is a valid configuration. Note that `neighborhood_iterator` may yield an invalid configuration.
Codecov Report
@@ Coverage Diff @@
## development #773 +/- ##
===============================================
- Coverage 87.02% 86.97% -0.06%
===============================================
Files 68 68
Lines 6221 6233 +12
===============================================
+ Hits 5414 5421 +7
- Misses 807 812 +5
Continue to review full report at Codecov.
|
Looks good to me! Thanks for the contribution! |
@dengdifan just wanted to make you aware that this only fixes symptoms that are caused by a bug in the ConfigSpace and should be fixed there. Therefore, once they are fixed, this change should be rolled back (and you should already create an issue for that ore something else to remember to undo these changes). |
* Documentation was updated thoroughly. A new theme with a new structure is provided and all pages have been updated. Also, the examples revised and up-to-date. * Option to use an own stopping strategy using `IncorporateRunResultCallback`. * Changed `scripts/smac` to `scripts/smac.py`. * `README.md` updated. * `CITATION.cff` added. * Made `smac-validate.py` consistent with runhistory and tae. (#762) * `minR`, `maxR` and `use_ta_time` can now be initialized by the scenario. (#775) * `ConfigSpace.util.get_one_exchange_neighborhood`'s invalid configurations are ignored. (#773) * Fixed an incorrect adaptive capping behaviour. (#749) * Avoid the potential `ValueError` raised by `LocalSearch._do_search`. (#773) Co-authored-by: dengdifan <[email protected]> Co-authored-by: Filip Bártek <[email protected]> Co-authored-by: Thomas Holvoet <[email protected]> Co-authored-by: benjamc <[email protected]> Co-authored-by: Carolin Benjamins <[email protected]> Co-authored-by: Marius Lindauer <[email protected]> Co-authored-by: eddiebergman <[email protected]> Co-authored-by: Difan Deng <[email protected]> Co-authored-by: dengdifan <[email protected]> Co-authored-by: Tim Ruhkopf <[email protected]>
The iterator returned by
ConfigSpace.util.get_one_exchange_neighborhood
may visit an invalid configuration. If it does so, it either raises aValueError
(with the probability 0.05) or yields the configuration (otherwise). This pull request ensures thatLocalSearch._do_search
handles these situations gracefully.ValueError
is raised, it is caught and ignored (instead of crashingsmac
).The script
run.sh
in demo.zip demonstrates raising ofValueError
from the iterator. The error is raised with a probability greater than 0 and smaller than 1.Issue at ConfigSpace that discusses the non-determinism of the configuration validation: automl/ConfigSpace#194