[Bug]: interaction between random seed and readparfile #643
Comments
|
I tried jittering without reading from the .par file and got different results each time. It looks like the optional user-input random seed is used in the creation of bootstrap data files here: https://github.com/nmfs-ost/ss3-source-code/blob/main/SS_write_ssnew.tpl#L22-L25 but not in the calculation of jittered initial values here: https://github.com/nmfs-ost/ss3-source-code/blob/main/SS_objfunc.tpl#L1488. If we were to use the user-controlled random seed for the jittered parameters then you would need to change the seed in the starter file each time your ran a new jitter. That's possible but might require some change in workflow. It might be tricky to use random seeds in the @MartaCousido, can you tell us more about the use case you have and how important setting the seed in the jitters would be? |
|
Thank you for taking the time to review this issue.
Perhaps I didn’t explain my use case clearly enough in my previous message.
I am not using the r4ss::jitter() function to perform jittering. Instead, I
am running the model multiple times, directly changing the seed in
starter.ss for each run.
The problem I’m experiencing is that, when I run the jitter in this way, it
seems that the model is not using the seed specified in starter.ss
correctly (I provide a different seed for each jittered model). I suspect
this because all the models return the same result, which is highly unusual
for a model as unstable as mine. This makes me think that it’s pulling the
seed from the system clock, and since all the models launch at the same
time, they end up with identical results.
To confirm this theory, I ran two tests:
Instead of specifying a different seed for each jittered model in
starter.ss, I left the seed field with -1 (no seed provided) and ran each
model at a different time. As I expected, of the 30 models, only 8 had the
same likelihood, indicating that something unusual happens when a fixed
seed is set in starter.ss.
I conducted a separate test from the jittering process: I ran three models
with the same seed in starter.ss and checked the results. The three models,
despite having the same seed, gave different results, which again suggests
that the model is using the system clock. Since these runs were started at
different times, they produced different outputs.
The seed is important to us because using the clock as the seed source
creates issues when we rely on a supercomputer with a queue system. In this
environment, even if models are launched at different times, they can begin
running simultaneously, which could result in identical outputs—not due to
model stability, but because they’ve used the same seed.
Thank you again for looking into this, and please let me know if further
clarification would be helpful.
Best regards,
Marta Cousido
El jue, 14 nov 2024 a las 1:10, Ian Taylor ***@***.***>)
escribió:
… I tried jittering without reading from the .par file and got different
results each time.
It looks like the optional user-input random seed is used in the creation
of bootstrap data files here:
https://github.com/nmfs-ost/ss3-source-code/blob/main/SS_write_ssnew.tpl#L22-L25
but not in the calculation of jittered initial values here:
https://github.com/nmfs-ost/ss3-source-code/blob/main/SS_objfunc.tpl#L1488
.
If we were to use the user-controlled random seed for the jittered
parameters then you would need to change the seed in the starter file each
time your ran a new jitter. That's possible but might require some change
in workflow. It might be tricky to use random seeds in the r4ss::jitter()
function which now has parallelization, so I think the easier solution
would just be to add an error if the user inputs a random seed. The benefit
would be more reproducibility in the jittering process. Alternatively we
could just document the fact that the seed only impacts the bootstrap data
files, not the jitters.
@MartaCousido <https://github.com/MartaCousido>, can you tell us more
about the use case you have and how important setting the seed in the
jitters would be?
—
Reply to this email directly, view it on GitHub
<#643 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ANOTIQYOCTKFYWHCXYPFYTD2APTATAVCNFSM6AAAAABRU32LDCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINZVGA3TQOJSG4>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
Thanks for the additional details @MartaCousido. @okenk and I thought about the possibility of simultaneous start times and matching seeds when jittering and thought that the "likelihood of the cores using the exact same seed is infinitesimal" https://r4ss.github.io/r4ss/reference/jitter.html#details. |
|
Hi Ian,
Thank you for looking into this issue and suggesting possible solutions.
As you point out, the probability of two models starting at exactly the
same time is indeed high. As mentioned in my previous email, this has
happened to us many times.
We raised this issue to explore the possibility of specifying a different
seed for each jittered model, independent of the start time. However, if
this isn't possible, we have implemented a workaround in our code.
Specifically, we've adapted the submission process to the supercomputer's
queuing system so that models are started at staggered times. Occasionally,
some models will still enter at the same time if they are held in the
queue, but in these cases we remove the duplicate model from the jitter
results.
Thanks again for your time and effort in the investigation of this issue.
El vie, 15 nov 2024 a las 17:52, Ian Taylor ***@***.***>)
escribió:
… Thanks for the additional details @MartaCousido
<https://github.com/MartaCousido>.
Your findings support my conclusion that the jitters is SS3 are using the
start time, not the random seed. But changing the code to make use of the
random seed could impact how users typically run jitters.
@okenk <https://github.com/okenk> and I thought about the possibility of
simultaneous start times and matching seeds when jittering and thought that
the "likelihood of the cores using the exact same seed is infinitesimal"
https://r4ss.github.io/r4ss/reference/jitter.html#details.
However, I now see that the C++ time() function used to set the seeds for
the jitters only provides the time in seconds, so simultaneous start times
could easily occur: https://en.cppreference.com/w/c/chrono/time.
One alternative solution might be to use a more precise clock using
something like
https://en.cppreference.com/w/cpp/chrono/high_resolution_clock.
Additionally the r4ss::jitter() function (if you're using it) could build
in some kind of random delay to the starting time to avoid simultaneous
starts.
—
Reply to this email directly, view it on GitHub
<#643 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ANOTIQ6Q3KW6ROEYDGJVUU32AYRGLAVCNFSM6AAAAABRU32LDCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINZZGQZDKMBQGU>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Describe the bug
During a jitter analysis (Marta Cousido) observed the following behaviour. When setting the seed in the starter file and using the "Initial Parameter Values=0" option, the seed seems to work correctly as I ran three models with the same seed and all three gave identical estimates.However, when I switch to "Initial Parameter Values=1" and set the seed in the starter file, the seed does not seem to work consistently. I ran three models with the same seed, but each gave different results. Is there any explanation for this behaviour?
To Reproduce
Rick will create a test example
Expected behavior
random seed to work as expected
Screenshots
No response
Which OS are you seeing the problem on?
No response
Which version of SS3 are you seeing the problem on?
No response
Additional Context
No response
The text was updated successfully, but these errors were encountered: