Degraded Performance in 0.6.0 vs. 0.5.0

[drpeterallan]

Hi, I've been running optimisations with an earlier version of pymoo (0.5.0) and have recently switched to the latest version (0.6.0) but I've noticed a drop in performance (see figure). If this may seem small its not and quite significant for my use case.

The only differences I've made to my code are how the sampling, crossover and mutation operators are defined, as per the new convention:

algorithm = NSGA3( pop_size=50, ref_dirs=ref_dirs, sampling=FloatRandomSampling(), cross_over=SBX(prob=0.5), mutation=PolynomialMutation(prob=0.5) )

I'm making the assumption that this is the same as previous, see here for old implementation:

algorithm = NSGA3( pop_size=50, ref_dirs=ref_dirs, sampling=get_sampling("real_random"), cross_over=get_crossover("real_sbx", prob=0.5), mutation=get_mutation("real_pm", prob=0.5) )

Firstly, is what I'm doing in the latest version the same as what I was doing previously, I've checked the documentation and it appears to be the case. If it is the same then do you have any ideas why the performance has dropped so much? I've ran few times changing the random seed and don't get back to the original performance, so I'm ruling out its just stochastic. I've also tweaked the population, crossover and mutation rates and got slight improvements but its still not as good as the older version.

Thanks in advance for your help and any suggestions would be welcome.

[bruscalia]

Hi, Peter!

I've tried to read the code in detail over these last few days to compare 0.5.0 to 0.6.* and I believe some keyword arguments in SBX and PM either became obsolete or changed their meaning.

prob in Mutation operators now seems to be a probability that each individual is mutated at all, and the old prob from PM now becomes prob_var (considering prob=1.0). The same has happened to SBX as the possibility of asymmetric crossovers is incorporated. Perhaps, that is the reason for the performance change in your problem.

The new features are interesting as they provide the user with more options on how the operators will work. However, I believe we must be concerned in future versions about changing the mechanisms related to some parameters and documenting their meaning in more detail. In my opinion, PM and SBX from 0.5.0 were easier to interpret and use than they are in version 0.6.* besides being more adherent to most studies in the literature on the subjects.

[drpeterallan]

Hi Bruno,

Many thanks for your help, I've changed both prob arguments in the mutation and crossover operators to prob_var and now performance is comparable with the older version. I've looked in the documentation and API reference and can't find any detailed description of these parameters. There's also no docstring in the code detailing the parameters. Which makes it difficult in understanding what exactly I'm doing and what these parameters should be set to.

Thanks again

Peter

[bruscalia]

You're welcome, Peter!

By the way, Julian started a discord chat with pymoo users (link available in pymoo home page). It would be nice to have you there, so we can share experiences using pymoo.