Exploring parallelism beyond futures.
The speaker wrote a parallelism library called claypoole, which
might be something useful to explore. Although, I don't know if we can
use it at iPlant or not.
(defn crps
"Continuous Rank Probability Score"
[precip-observed precip-model]
(->> (map single-crps precip-observed precip-model)
(reduce +)
;;
(* @scale)))You'll have problems if:
- Your tasks are small compared to the overhead
- You want to control the number of concurrent threads.
- you expect exceptions to work normally.
- Lazy and needs to be driven.
- Generates threads as needed.
- Beware of simultaneous pmaps.
- It's wacky when there's chunking.
- Runs roughly ncpus + 3 tasks.
- A long-running task will stall it.
- All limitations of futures.
- Uses CSP channels and coroutines.
- Looks kind of like Go.
- Reads like one flow: avoid callback hell.
- Uses cooperative multithreading.
- Backed by a fixed-size thread pool.
- Mostly for concurrentcy, not parallelism
- You shouldn't block too many of its threads.
- Easy to wait on other work.
- Use it to interact with worker threads.
- Parallelism using
pipeline- Runs a transducer between two channels with parallelism
n - Also
pipeline-asyncandpipeline-blocking
- Runs a transducer between two channels with parallelism
- Exceptions will kill your coroutines.
https://github.com/TheClimateCorporation/claypoole
- Uses thread pools to control parallelism.
- Can auto-manage thread pools.
- Tries to get things done fast.
- Default is eager.
- Ouptut blocks on incomplete tasks.
- Doesn't stall on slow tasks.
- Streaming seqs can be chained.
future,pmap,pfor- Unordered functions.
- Lazy versions of these functions are available.
- Exceptions are re-thrown correctly.
- Eliminates chunking.
- Priorities are available.
https://github.com/aphyr/tesser
How might we use some of these libraries at iPlant?