One-exchange neighborhood may raise ValueError non-deterministically

[filipbartek]

Description

ConfigSpace.util.get_one_exchange_neighbourhood uses np.random to decide whether a new configuration is to be validated (see util.pyx:218). Using np.random for this decision makes the behavior of the function non-deterministic.

Solution proposal

Use the randomness generator random instantiated at util.pyx:130 instead of np.random.

[mfeurer]

Thanks a lot for reporting this. Would you mind creating a pull request fixing this? What worries me a bit more, though, is that this is actually triggered - this should not happen at all as the neighborhood generation should only return valid configurations. Do you have a minimal example that reproduces this issue?

[filipbartek]

Minimal example

Try to run the following PCS with test_sample_configuration_spaces.py: vampire.zip

If all the calls to cs._check_configuration_rigorous are removed from the test body, a ValueError is raised from n.is_valid_configuration() at "test_sample_configuration_spaces.py:74":

E   ValueError: Active hyperparameter 'age_weight_ratio:log_ratio' not specified!

If n.is_valid_configuration() is removed from the test body as well, a ValueError is raised non-deterministically from enumerate(neighborhood) (line 73).

Note that if the calls to cs._check_configuration_rigorous are left in place, a different ValueError is raised from cs._check_configuration_rigorous(default) at "test_sample_configuration_spaces.py:52":

E   ValueError: Inactive hyperparameter 'age_weight_ratio:log_ratio' must not be specified, but has the vector value: '0.7692307692307693'.

[mfeurer]

Thanks a lot for the PR and the additional example. So apparently there is an issue in the neighborhood generation that leads to the creation of invalid configurations. If you are familiar with cython and can have a look in there this would be very much appreciated.

[mfeurer]

Re-opening this as #195 only solves half of this issue.

[filipbartek]

So apparently there is an issue in the neighborhood generation that leads to the creation of invalid configurations. If you are familiar with cython and can have a look in there this would be very much appreciated.

I am investigating the issue. It seems that ConfigSpace.c_util.change_hp_value invoked from get_one_exchange_neighbourhood may return an invalid configuration if a hyperparameter is guarded by a condition of type OrConjunction. The docstring of change_hp_value suggests that the function may return an invalid configuration:

Does not check if the new value is legal. Activates and deactivates other
hyperparameters if necessary. Does not check if new hyperparameter value
results in the violation of any forbidden clauses.

@mfeurer, should change_hp_value in fact guarantee that the output configuration is valid, or is checking the validity a responsibility of the caller (get_one_exchange_neighbourhood)?

[filipbartek]

#197 modifies change_hp_value so that it produces a valid configuration in each of the calls invoked by the minimal example described above. Note that the proposed solution probably degrades performance in some cases.

[eddiebergman]

I can confirm that this issue still exists in 1.0.0 and is likely related to the || condition above.

I've created an easier repro example:

repro = r"""
age_weight_ratio:log_ratio real [-10.0, 3.0] [0.0]
saturation_algorithm categorical {discount, fmb, inst_gen, lrs, otter, z3} [lrs]
inst_gen_with_resolution categorical {off, on} [off]

age_weight_ratio:log_ratio | saturation_algorithm != inst_gen || inst_gen_with_resolution == on
inst_gen_with_resolution | saturation_algorithm == inst_gen
""".splitlines()

from ConfigSpace.read_and_write.pcs_new import read

cs = read(repro)
default = cs.get_default_configuration()
print(default)
cs._check_configuration_rigorous(default)