Parallelizing Analysis using PyLOOS and LOOS

[krb0073]

Hi All,
I have been working on a package to collect 30+ time series measurements on a selection. I have a good serial python code that reads a configuration and out puts a file for each.I was wondering, what would be the best practice to analyze trajectory data concurrently?
Thanks for the help,
Kyle

[agrossfield]

In my experience, the best parallelization tool for analysis is the ampersand! Assuming you've got 30 separate jobs right now, you probably already have a script to run the analyses, so you can just launch them in parallel, in groups of 4 or 8, depending on how many cores you have, how fast your filesystem is, and the relative cost of the compute and i/o portions of your analysis.

One dumb but effective way to do it is something like this:

num_jobs_at_once = 4
for i in range(num_jobs):
    # do something to build up the command line programmatically
    command_line = "string that runs the analysis command"
    if i % num_jobs_at_once != 0:
        command_line += " &"
    system(command_line)

This is far from bullet-proof (and like I said, kind of dumb), but it'll run 3 jobs in the background, then wait for the 4th to complete. Assuming the jobs take roughly the same amount of time, it'll keep you from slamming your system by launching 30 jobs at once.

[krb0073]

That is actually really helpful for another project that I am working with. After some digging I found that this worked well to run a function over parallel from within the python scripts.

import multiprocessing as mp
# variable list will hold a list of tuples which each inner list defines the variables to use per run 
var_list = []
A = [5] *10
B = [x for x in range(11)]
C = [x  -1 for x in range(11)]
def product(a,b,c):
    return a* 2b * c
pool = mp.Pool(processes=n_cores) # make a object # get the pool of processors used
ans = pool.starmap(product,zip(A,B,C))
pool.close() # release the cores for now  
pool.join() # Blocks until all items in the queue have been gotten and processed

it is not super clean but it allows me to a function run on any number of processors. I chose this because I built a class that had all different analysis that i do within and i did not feel like writing all the functions i had in the class to different files.