data management

[berceanu]

Not sure if this issue belongs here or in the upstream libEnsemble.

When doing manual parameter scans in the past, I encountered the need for a data management framework, in order to store, organize and later post-process the output openPMD files. This led to me couple the fbpic code with the signac framework, as shown here. Furthermore, the use of signac-flow allows one to couple multiple codes, like using the PIC output as input for Geant4 etc.

My question is, is this something you plan to have in the scope of optimas? My understanding is that libEnsemble's history array is much more limited in this regard.

[jmlarson1]

Hi @berceanu. In my mind, the libE history array is quite flexible as long as you know the size of the entry for each simulation output. If the collective outputs from the simulations is too large to consider storing in one place, I would recommend saving each output to a file (e.g., named with a UUID) and then putting the name of the file in the history array.

Is there some additional information that is desired in the libE history array that isn't currently available?

[berceanu]

Hi @jmlarson1. Well, usually each PIC simulation code outputs hundreds of .hfd5 files per run, which need to be post-processed in order to extract, for instance, the electron trajectories to compute the emitted radiation, or to make a movie of the electron bunch evolution in real/phase space.

The point is also that sometimes in long-running simulations you don't always know a priori all the things you want to analyze, so it can be useful the keep the data around.

[jmlarson1]

Hi @berceanu. I certainly understand that. We've tried to design libEnsemble (and optimas) to allow for all simulation-output to be stored in a (relatively) easy to retrieve manner.

For example, libEnsemble can be configured so that every simulation occurs in its own directory (with a name that includes the sim_id). For your used case, I'd recommend running in such a manner. Then after the run, you can look through the history array for the sim_id that you like to investigate further.

Would that work for you?

[berceanu]

I see. How about the workflow management part, for instance, if one wants to join together two different codes, and use the output of one code as input for another, but only launching the second code once the first one finished successfully?

[AngelFP]

Hi @berceanu, as mentioned by @jmlarson1 each simulation carried out in optimas generates it's own directory. All the output (e.g., the hdf5 diagnostics) will be stored there. You will typically define a function that analyzes this output to determine the value of your optimization objectives or any other diagnostics you are interested in. You can check an example on how to do this with FBPIC here.

Regarding your last point, are you running a study where each simulation needs to run two codes? (e.g., an FBPIC simulation, followed by another with ASTRA, Geant4, etc where you use the beam produced by FBPIC?) If so, what you would could do right now is create a template_simulation_script.py that runs one code after the other.

It's not an ideal solution, as by doing this there is no obvious way of running each simulation with different resources (e.g., MPI processes). It's a good point that we could consider for the future. Basically being able to have several "steps" per evaluation, each being a different simulation that might require different resources.

[jmlarson1]

A follow-on question for @berceanu . Are the resource requirements for both of the two codes large but different? If one is "very cheap", then there should be no need to worry about resources.

But if both codes need considerable compute resources, but one needs many CPUs (but no GPUs) and the other is the opposite, then it's certainly an interesting use case to consider supporting.

[berceanu]

@AngelFP and @jmlarson1, thank you both for the follow-up. Indeed, the scenario I have in mind consists of joining together a GPU-intensive PIC code, followed by a CPU-intensive radiation code, so the resource allocation would look quite different for the two.

[AngelFP]

Ok, sounds good. I think this is a feature we should implement. I'll have a look at how this could be done, but I can't promise any ETA at the moment.

[jmlarson1]

@berceanu We've had a few discussions of this and many questions were raised

1. Do you imagine all "completed GPU" run would always start a "CPU run"?
2. Is there any concern about the efficient resource utilization? Say every GPU-run starts a CPU-run and each GPU-run and CPU-run takes the same time. Then half of the time, the GPUs will be idle and the other half of the time the CPUs will be idle.

[AngelFP]

@jmlarson1 Probably all we would need is a sim_f where we call executor.submit several times by, for example, looping over a series of steps that correspond to the individual simulations and their settings. We would pass these steps in the sim_specs['user']. And I guess that every step should register an application.

[berceanu]

Perhaps it would help if we make this a bit more concrete by picking some specific scenarios.
Lets's say one runs Wake-T, or a PIC code (or both) for a simple electron acceleration stage and then uses a second code like Geant4 to compute the radiation generated by the accelerated electron bunch as it collides with a solid target, for instance. In that case, one could have a global optimization objective, like the total number of emitted photons (at a certain energy) from Geant4.
Alternatively, if we consider a betatron setup, one could run a PIC code in the first stage, have a second stage that extracts the electron trajectories from the PIC output, and a third and final stage that computes the synchrotron radiation emitted by those electrons. Again, the optimization objective could be related to the final synchrotron spectrum.
In both these scenarios, there could be early stopping conditions for the first step. Let's say the system is unstable for the laser/plasma parameters chosen by the optimizer and you lose all electrons. So a condition like if Q < x pC (for some value of x) would skip the next stages in the optimization process for that run.
A third scenario would be combining a CPU-intensive hydrodynamics code like FLASH with a PIC code like WarpX for modelling TNSA/RPA and including prepulse effects.

[berceanu]

Any feedback on this?

[jmlarson1]

I'm certainly interested in helping support this use case. Do you have an example script (or scripts) that performs these three calls?

1. PIC + Geant4
2. PIC+ Radiation
3. FLASH+PIC
   ?

[AngelFP]

Same here, I think we should support this type of workflow. This could probably be implemented by allowing the user to create several Evaluators (one per simulation code/step) that can be chained and executed in series. I don't have the bandwidth to look at this in the next days, but I'll try to do so sooner than later. As Jeff says, any dummy example scripts/workflow you could share would be very useful.

[AngelFP]

From the user side, we could support this workflow by allowing something like this:

# Create one evaluator per simulation code
ev_fbpic = TemplateEvaluator(
    sim_template='template_simulation_fbpic.py',
    analysis_func=analyze_simulation_fbpic,
    n_gpus=2
)
ev_g4 = TemplateEvaluator(
    sim_template='template_simulation_geant4.txt',
    analysis_func=analyze_simulation_g4,
    executable='geant4',
    n_procs=40
)

# Create exploration.
exp = Exploration(
    generator=gen,
    evaluator=[ev_fbpic, ev_g4],
    max_evals=100,
    sim_workers=4
)

The user would pass a list of evaluators to the exploration in the order in which they should run. Each evaluator can have its own resources and analysis function (e.g., in this example analyze_simulation_fbpic could prepare a particle distribution to be read by Geant4).

Any comments/ideas on this?

[jmlarson1]

This seems reasonable to me. The underlying libE executor should be able to handle this.

[AngelFP]

Hi @berceanu, I gave it a try and implemented something very similar to the solution above (see #110). Would you be able to try this branch and let us know if it works well for your use case?

[berceanu]

Hi @AngelFP that is great, can't wait to test it out! It might take a while though, since I am currently on holiday.

[AngelFP]

Let us know if you find any issues once you get to try it. Enjoy the holidays!

[berceanu]

By the way, speaking of combining multiple codes such as PIC + particle tracking, I recently wrote an openPMD plugin for particle downsampling for such a usecase, here is the repo in case you find it useful for your workflows.

[AngelFP]

Thanks for sharing, that could indeed be useful in some cases. Did the solution with the ChainEvaluator work for you?

[berceanu]

I will report back as soon as I try it out, didn't get the chance yet.

[AngelFP]

Hi @berceanu, any updates on this? I've been using the ChainEvaluator myself without problems, so I'll be closing this issue. Feel free to reopen it if it somehow does not work in your case.