diff --git a/404.html b/404.html index 692b177..8ab62d7 100644 --- a/404.html +++ b/404.html @@ -7,10 +7,10 @@ Page not found (404) • clustermq - - + + - + License • clustermqLicense • clustermq @@ -10,7 +10,7 @@ clustermq - 0.8.96 + 0.9.0 + + + + + +
+
+
+

Class for basic queuing system functions

+ Source: R/pool.r +
Pool.Rd
+
+ +
+

Provides the basic functions needed to communicate between machines +This should abstract most functions of rZMQ so the scheduler +implementations can rely on the higher level functionality

+
+ + + +
+ + +
+ + + + + + + diff --git a/reference/Q.html b/reference/Q.html index 5934146..ee32652 100644 --- a/reference/Q.html +++ b/reference/Q.html @@ -1,5 +1,5 @@ -Queue function calls on the cluster — Q • clustermqQueue function calls on the cluster — Q • clustermq @@ -10,7 +10,7 @@ clustermq - 0.8.96 + 0.9.0 - - - - - -
-
-
-

Wrap C++ Rcpp module in R6 to get reliable argument matching

- Source: R/zeromq.r -
ZeroMQ.Rd
-
- -
-

This is an R6 wrapper of the C++ class in order to support R argument -matching. Ideally, Rcpp will at some point support this natively and this -file will no longer be necessary. Until then, it causes redundancy with -zeromq.cpp, but this is a small inconvenience and much less error-prone than -only relying on positional arguments.

-
- - - -
- - -
-

Developed by Michael Schubert.

-
- -
-

Site built with pkgdown 2.0.2.

-
- -
- - - - - - - diff --git a/reference/bind_avail.html b/reference/bind_avail.html deleted file mode 100644 index e26f329..0000000 --- a/reference/bind_avail.html +++ /dev/null @@ -1,104 +0,0 @@ - -Binds a ZeroMQ socket to an available port in given range — bind_avail • clustermq - Skip to contents - - -
-
-
-

Binds a ZeroMQ socket to an available port in given range

- Source: R/util.r -
bind_avail.Rd
-
- -
-

Binds a ZeroMQ socket to an available port in given range

-
- -
-

Usage

- 
bind_avail(socket, range, iface = "tcp://*", n_tries = 100)
-
- -
-

Arguments

-
socket
-

An ZeroMQ socket object

-
range
-

Numbers to consider (e.g. 6000:8000)

-
iface
-

Interface to listen on

-
n_tries
-

Number of ports to try in range

-
-
-

Value

-

The port the socket is bound to

-
- -
- - -
-

Developed by Michael Schubert.

-
- -
-

Site built with pkgdown 2.0.2.

-
- -
- - - - - - - diff --git a/reference/check_args.html b/reference/check_args.html index 63a52ab..33fed69 100644 --- a/reference/check_args.html +++ b/reference/check_args.html @@ -1,5 +1,5 @@ -Function to check arguments with which Q() is called — check_args • clustermqFunction to check arguments with which Q() is called — check_args • clustermq @@ -10,7 +10,7 @@ clustermq - 0.8.96 + 0.9.0 + + + + + +
+
+
+

Message format for logging

+ Source: R/util.r +
msg_fmt.Rd
+
+ +
+

Message format for logging

+
+ +
+

Usage

+ 
msg_fmt(verbose = TRUE)
+
+ + +
+ + +
+

Developed by Michael Schubert, ZeroMQ authors.

+
+ +
+

Site built with pkgdown 2.0.7.

+
+ +
+ + + + + + + diff --git a/reference/purrr_lookup.html b/reference/purrr_lookup.html deleted file mode 100644 index 6fac8ad..0000000 --- a/reference/purrr_lookup.html +++ /dev/null @@ -1,93 +0,0 @@ - -Lookup table for return types to purrr functions — purrr_lookup • clustermq - Skip to contents - - -
-
-
-

Lookup table for return types to purrr functions

- Source: R/util.r -
purrr_lookup.Rd
-
- -
-

Lookup table for return types to purrr functions

-
- -
-

Usage

- 
purrr_lookup
-
- -
-

Format

-

An object of class list of length 9.

-
- -
- - -
-

Developed by Michael Schubert.

-
- -
-

Site built with pkgdown 2.0.2.

-
- -
- - - - - - - diff --git a/reference/register_dopar_cmq.html b/reference/register_dopar_cmq.html index f4ed2a1..ba701c4 100644 --- a/reference/register_dopar_cmq.html +++ b/reference/register_dopar_cmq.html @@ -1,5 +1,5 @@ -Register clustermq as `foreach` parallel handler — register_dopar_cmq • clustermqRegister clustermq as `foreach` parallel handler — register_dopar_cmq • clustermq @@ -10,7 +10,7 @@ clustermq - 0.8.96 + 0.9.0 + + + + + +
+
+
+

Wraps an error in a condition object

+ Source: R/util.r +
wrap_error.Rd
+
+ +
+

Wraps an error in a condition object

+
+ +
+

Usage

+ 
wrap_error(call)
+
+ + +
+ + +
+

Developed by Michael Schubert, ZeroMQ authors.

+
+ +
+

Site built with pkgdown 2.0.7.

+
+ +
+ + + + + + + diff --git a/search.json b/search.json index 98254ab..97d12bc 100644 --- a/search.json +++ b/search.json @@ -1 +1 @@ -[{"path":"/articles/quickstart.html","id":"installation","dir":"Articles","previous_headings":"","what":"Installation","title":"Quick Start","text":"First, need ZeroMQ system library. probably already installed system. , package manager provide : install clustermq package R CRAN: Alternatively can use remotes package install directly Github: good go! default, clustermq look sbatch (SLURM), bsub (LSF), qsub (SGE) $PATH use scheduler available. examples don’t run box, might need set scheduler explicitly.","code":"# You can skip this step on Windows and macOS, the package binary has it # On a computing cluster, we recommend to use Conda or Linuxbrew brew install zeromq # Linuxbrew, Homebrew on macOS conda install zeromq # Conda, Miniconda sudo apt-get install libzmq3-dev # Ubuntu sudo yum install zeromq-devel # Fedora pacman -S zeromq # Arch Linux install.packages('clustermq') # install.packages('remotes') remotes::install_github('mschubert/clustermq') # remotes::install_github('mschubert/clustermq', ref=\"develop\") # dev version"},{"path":"/articles/quickstart.html","id":"setting-up-the-scheduler-explicitly","dir":"Articles","previous_headings":"","what":"Setting up the scheduler explicitly","title":"Quick Start","text":"HPC cluster’s scheduler ensures computing jobs distributed available worker nodes. Hence, clustermq interfaces order computations. currently support following schedulers (either locally via SSH): Multiprocess - test calls parallelize cores using options(clustermq.scheduler=\"multiprocess\") LSF - work without setup SGE - work without setup SLURM - work without setup PBS/Torque - needs options(clustermq.scheduler=\"PBS\"/\"Torque\") via SSH - needs options(clustermq.scheduler=\"ssh\", clustermq.ssh.host=) Default submission templates provided can customized, e.g. activate compute environments containers.","code":""},{"path":"/articles/quickstart.html","id":"examples","dir":"Articles","previous_headings":"","what":"Examples","title":"Quick Start","text":"package designed distribute arbitrary function calls HPC worker nodes. , however, couple caveats observe R session running worker share local memory. simplest example function call completely self-sufficient, one argument (x) iterate : Non-iterated arguments supported const argument: function relies objects environment passed arguments, can exported using export argument: want use package function need load worker using library() call referencing package_name::: clustermq can also used parallel backend foreach. also used BiocParallel, can run packages cluster well: examples available user guide.","code":"fx = function(x) x * 2 Q(fx, x=1:3, n_jobs=1) #> Running sequentially ('LOCAL') ... #> [[1]] #> [1] 2 #> #> [[2]] #> [1] 4 #> #> [[3]] #> [1] 6 fx = function(x, y) x * 2 + y Q(fx, x=1:3, const=list(y=10), n_jobs=1) #> Running sequentially ('LOCAL') ... #> [[1]] #> [1] 12 #> #> [[2]] #> [1] 14 #> #> [[3]] #> [1] 16 fx = function(x) x * 2 + y Q(fx, x=1:3, export=list(y=10), n_jobs=1) #> Running sequentially ('LOCAL') ... #> [[1]] #> [1] 12 #> #> [[2]] #> [1] 14 #> #> [[3]] #> [1] 16 fx = function(x) { `%>%` = dplyr::`%>%` x %>% dplyr::mutate(area = Sepal.Length * Sepal.Width) %>% head() } Q(fx, x=list(iris), n_jobs=1) #> Running sequentially ('LOCAL') ... #> [[1]] #> Sepal.Length Sepal.Width Petal.Length Petal.Width Species area #> 1 5.1 3.5 1.4 0.2 setosa 17.85 #> 2 4.9 3.0 1.4 0.2 setosa 14.70 #> 3 4.7 3.2 1.3 0.2 setosa 15.04 #> 4 4.6 3.1 1.5 0.2 setosa 14.26 #> 5 5.0 3.6 1.4 0.2 setosa 18.00 #> 6 5.4 3.9 1.7 0.4 setosa 21.06 library(foreach) register_dopar_cmq(n_jobs=2, memory=1024) # accepts same arguments as `workers` foreach(i=1:3) %dopar% sqrt(i) # this will be executed as jobs #> Running sequentially ('LOCAL') ... #> [[1]] #> [1] 1 #> #> [[2]] #> [1] 1.414214 #> #> [[3]] #> [1] 1.732051 library(BiocParallel) register(DoparParam()) # after register_dopar_cmq(...) bplapply(1:3, sqrt)"},{"path":"/articles/quickstart.html","id":"usage","dir":"Articles","previous_headings":"","what":"Usage","title":"Quick Start","text":"following arguments supported Q: fun - function call. needs self-sufficient (access master environment) ... - iterated arguments passed function. one, need named const - named list non-iterated arguments passed fun export - named list objects export worker environment Behavior can fine-tuned using options : fail_on_error - Whether stop one calls returns error seed - common seed combined job number reproducible results memory - Amount memory request job (bsub -M) n_jobs - Number jobs submit function calls job_size - Number function calls per job. used combination n_jobs latter overall limit chunk_size - many calls worker process reporting back master. Default: every worker report back 100 times total full documentation available typing ?Q.","code":""},{"path":"/articles/quickstart.html","id":"comparison-to-other-packages","dir":"Articles","previous_headings":"","what":"Comparison to other packages","title":"Quick Start","text":"packages provide high-level parallelization R function calls computing cluster. thorough comparison features performance available wiki. Briefly, compare long takes different HPC scheduler tools submit, run collect function calls negligible processing time (multiplying numeric value 2). serves quantify maximum throughput can reach BatchJobs, batchtools clustermq. find BatchJobs unable process 106 calls produces reproducible RSQLite error. batchtools able process function calls, file system practically limits 106 calls. clustermq problems processing 109 calls, still faster batchtools 106 calls. short, use ClusterMQ want: one-line solution run cluster jobs minimal setup access cluster functions local Rstudio via SSH fast processing many function calls without network storage /O Use batchtools : want use mature well-tested package don’t mind arguments every call written /read disc don’t mind ’s load-balancing run-time Use Snakemake (flowr, remake, drake) : want design run pipeline different tools Don’t use batch (last updated 2013) BatchJobs (issues SQLite network-mounted storage).","code":""},{"path":[]},{"path":"/articles/technicaldocs.html","id":"base-api-and-schedulers","dir":"Articles","previous_headings":"Worker API","what":"Base API and schedulers","title":"Technical Documentation","text":"main worker functions wrapped R6 class name QSys. provides standardized API lower-level messages sent via ZeroMQ. base class derived scheduler classes add required functions submitting cleaning jobs: pool workers can created using workers() function, instantiates object corresponding QSys-derived scheduler class. See ?workers details.","code":"+ QSys |- Multicore |- LSF + SGE |- PBS |- Torque |- etc. # start up a pool of three workers using the default scheduler w = workers(n_jobs=3) # if we make an unclean exit for whatever reason, clean up the jobs on.exit(w$finalize())"},{"path":"/articles/technicaldocs.html","id":"worker-startup","dir":"Articles","previous_headings":"Worker API","what":"Worker startup","title":"Technical Documentation","text":"workers started via scheduler, know machine run . start every worker TCP/IP address master socket distribute work. achieved call R common schedulers: master’s side, wait worker connects:","code":"R --no-save --no-restore -e 'clustermq:::worker(\"{{ master }}\")' # this will block until a worker is ready msg = w$receive_data()"},{"path":"/articles/technicaldocs.html","id":"common-data-and-exports","dir":"Articles","previous_headings":"Worker API","what":"Common data and exports","title":"Technical Documentation","text":"Workers start without knowledge process . order transfer initial data worker, first create serialize list object following fields: fun - function call iterated data const - constant data function call receive export - objects exported workers’ .GlobalEnv rettype - character string data type return; e.g. list, logical common_seed - random seed function calls; offset job ID token - character string identify data set; optional, automatically generated token returned none given Workers connect master send list field token. can used check worker already received common data supposed work .","code":"# create a reusable, serialized ZeroMQ object with the common data on the master w$set_common_data(fun, const, export, rettype, common_seed, token) if (msg$token != ) w$send_common_data()"},{"path":"/articles/technicaldocs.html","id":"iterated-data","dir":"Articles","previous_headings":"Worker API","what":"Iterated data","title":"Technical Documentation","text":"worker already received common data, can send chunk iterated arguments work . passed list iterables, e.g. data.frame column iterated argument. also needs column name id, used identify call. worker finished processing, send message field result list, containing: result - named rettype results warnings - list warning messages individual calls errors - list error messages individual calls","code":"chunk = data.frame(arg1=1:5, arg2=5:1, ` id `=1:5) w$send_job_data(chunk) msg = w$receive_data() if (!is.null(msg$result)) { # store result here, handle errors/warnings if required }"},{"path":"/articles/technicaldocs.html","id":"custom-calls","dir":"Articles","previous_headings":"Worker API","what":"Custom calls","title":"Technical Documentation","text":"Apart sending common iterated data worker process chunks, also possible send arbitrary calls evaluate. needs following fields: expr - expression evaluated env - list additional objects required perform call ref - identifier call; default expression ","code":"w$send_call(expr, env=list(...), ref=\"mycall1\")"},{"path":"/articles/technicaldocs.html","id":"main-event-loop","dir":"Articles","previous_headings":"Worker API","what":"Main event loop","title":"Technical Documentation","text":"Putting together event loop, get essentially implemented master. loop similar structure can used extend clustermq. example, done drake using common data custom calls (iterated chunks).","code":"w = workers(3) on.exit(w$finalize()) while (we have new work to send) { msg = w$receive_data() if (!is.null(msg$result)) # handle result if (msg$token != ) w$send_common_data() else w$send_job_data(...) } # if proper cleanup is successful, cancel kill-on-exit if (w$cleanup()) on.exit()"},{"path":"/articles/technicaldocs.html","id":"zeromq-message-specification","dir":"Articles","previous_headings":"","what":"ZeroMQ message specification","title":"Technical Documentation","text":"Communication master (main event loop) workers (QSys base class) organised messages. chunks serialized data id field, data required type message.","code":""},{"path":"/articles/technicaldocs.html","id":"messages-per-type","dir":"Articles","previous_headings":"ZeroMQ message specification","what":"Messages per type","title":"Technical Documentation","text":", message id listed additional fields per message.","code":""},{"path":"/articles/technicaldocs.html","id":"worker","dir":"Articles","previous_headings":"ZeroMQ message specification > Messages per type","what":"Worker","title":"Technical Documentation","text":"workflow handled worker() event loop clustermq (confused workers() control). function called every job thread interact master(). event loop internal, .e. modifiable exported.","code":""},{"path":"/articles/technicaldocs.html","id":"worker_up","dir":"Articles","previous_headings":"ZeroMQ message specification > Messages per type > Worker","what":"WORKER_UP","title":"Technical Documentation","text":"Message ID indicating worker accepting data Field worker_id master empty ssh_proxy Answer serialized common data (fun, const, seed) redirect (URL worker can get data)","code":""},{"path":"/articles/technicaldocs.html","id":"worker_ready","dir":"Articles","previous_headings":"ZeroMQ message specification > Messages per type > Worker","what":"WORKER_READY","title":"Technical Documentation","text":"Message ID indicating worker accepting chunks may contain field result finished chunk processing failed, result object type error result named rettype results warnings list warning messages individual calls errors list error messages individual calls","code":""},{"path":"/articles/technicaldocs.html","id":"worker_done","dir":"Articles","previous_headings":"ZeroMQ message specification > Messages per type > Worker","what":"WORKER_DONE","title":"Technical Documentation","text":"Message ID indicating worker shutting Worker send response WORKER_STOP Field time (Sys.time()), mem (max memory used) calls (number processed calls)","code":""},{"path":"/articles/technicaldocs.html","id":"worker_error","dir":"Articles","previous_headings":"ZeroMQ message specification > Messages per type > Worker","what":"WORKER_ERROR","title":"Technical Documentation","text":"error occurred processing flow (function calls ) Field msg describing error Master shut receiving signal","code":""},{"path":"/articles/technicaldocs.html","id":"master","dir":"Articles","previous_headings":"ZeroMQ message specification > Messages per type","what":"Master","title":"Technical Documentation","text":"workflow handled master() function clustermq. using Q() Q_rows(), handled hood. Workers created outside functions can reused within Q()/Q_rows() without knowing internal message structure. documentation show possible implement custom control flow, e.g. want evaluate arbitrary expressions workers instead defining one function call different arguments.","code":"w = workers(n_jobs, ...) # w$cleanup() for a clean shutdown at the end"},{"path":"/articles/technicaldocs.html","id":"do_setup","dir":"Articles","previous_headings":"ZeroMQ message specification > Messages per type > Master","what":"DO_SETUP","title":"Technical Documentation","text":"Message contains common data, like function call arguments Required fields : fun, const, export, rettype, common_seed, token Worker respond WORKER_READY","code":"# create a reusable, serialized ZeroMQ object with the common data on the master w$set_common_data() # send this object to a worker w$send_common_data()"},{"path":"/articles/technicaldocs.html","id":"do_chunk","dir":"Articles","previous_headings":"ZeroMQ message specification > Messages per type > Master","what":"DO_CHUNK","title":"Technical Documentation","text":"Chunk iterated arguments worker Field chunk, data.frame row call columns arguments Row names chunk used call IDs","code":"w$send_job_data()"},{"path":"/articles/technicaldocs.html","id":"do_call-new-in-0-8-5","dir":"Articles","previous_headings":"ZeroMQ message specification > Messages per type > Master","what":"DO_CALL (new in 0.8.5)","title":"Technical Documentation","text":"Evaluate specific expression worker Needs fields expr (expression evaluated) env (list environment evaluate )","code":"w$send_call()"},{"path":"/articles/technicaldocs.html","id":"worker_wait","dir":"Articles","previous_headings":"ZeroMQ message specification > Messages per type > Master","what":"WORKER_WAIT","title":"Technical Documentation","text":"Instruct worker wait wait seconds Worker respond WORKER_READY","code":"w$send_wait()"},{"path":"/articles/technicaldocs.html","id":"worker_stop","dir":"Articles","previous_headings":"ZeroMQ message specification > Messages per type > Master","what":"WORKER_STOP","title":"Technical Documentation","text":"Instruct worker exit main event loop message fields","code":"w$send_shutdown_worker()"},{"path":"/articles/technicaldocs.html","id":"disconnect-and-reset-socket-state","dir":"Articles","previous_headings":"ZeroMQ message specification > Messages per type > Master","what":"Disconnect and reset socket state","title":"Technical Documentation","text":"","code":"w$disconnect_worker()"},{"path":"/articles/technicaldocs.html","id":"control-flow-stages","dir":"Articles","previous_headings":"ZeroMQ message specification","what":"Control flow stages","title":"Technical Documentation","text":"convention master > worker","code":""},{"path":"/articles/technicaldocs.html","id":"batch-processing-no-proxy","dir":"Articles","previous_headings":"ZeroMQ message specification > Control flow stages","what":"Batch processing, no proxy","title":"Technical Documentation","text":"default use case Q, Q_rows. set common data (DO_SETUP; function call, constant arguments, exported data, random seed) provide chunks arguments (DO_CHUNK) data.frames batch processing. DO_SETUP DO_CHUNK [repeat] WORKER_STOP WORKER_DONE can implemented following way:","code":"w$set_common_data(...) while(work remaining or w$workers_running > 0) { msg = w$receive_data() # wait until a worker is ready if (msg$id == \"WORKER_UP\") { # treat same as WORKER_READY if no common data w$send_common_data() } else if (msg$id == \"WORKER_READY\") { if (work remaining) w$send_job_data(data.frame with arguments for all calls of this chunk) else w$send_shutdown_worker() # ..handle your result.. } else if (msg$id == \"WORKER_DONE\") { w$disconnect_worker() } else if (msg$id == \"WORKER_ERROR\") { stop(\"processing error\") } }"},{"path":"/articles/technicaldocs.html","id":"evaluating-custom-expressions","dir":"Articles","previous_headings":"ZeroMQ message specification > Control flow stages","what":"Evaluating custom expressions","title":"Technical Documentation","text":"can mixed batch processing, long DO_SETUP called DO_CHUNK (otherwise cause WORKER_ERROR token mismatch). DO_SETUP DO_CALL (e.g. export commonly used data) DO_CALL [repeat] WORKER_STOP WORKER_DONE can implemented following way:","code":"w$set_common_data(...) # optional, if common data required while(work remaining or w$workers_running > 0) { msg = w$receive_data() # wait until a worker is ready if (msg$id == \"WORKER_UP\") { # treat same as WORKER_READY if no common data w$send_common_data() } else if (msg$id == \"WORKER_READY\") { if (work remaining) w$send_call(expr, env) else w$send_shutdown_worker() # ..handle your result.. } else if (msg$id == \"WORKER_DONE\") { w$disconnect_worker() } else if (msg$id == \"WORKER_ERROR\") { stop(\"processing error\") } }"},{"path":[]},{"path":"/articles/userguide.html","id":"zeromq","dir":"Articles","previous_headings":"Installation","what":"ZeroMQ","title":"User Guide","text":"First, need ZeroMQ system library. probably already installed system. , package manager provide : can also manually install release, e.g. using zeromq-4.3.3.tar.gz: R ZeroMQ provided package manager need let R know find library installing clustermq:","code":"# You can skip this step on Windows and macOS, the package binary has it # On a computing cluster, we recommend to use Conda or Linuxbrew brew install zeromq # Linuxbrew, Homebrew on macOS conda install zeromq # Conda, Miniconda sudo apt-get install libzmq3-dev # Ubuntu sudo yum install zeromq-devel # Fedora pacman -S zeromq # Arch Linux # tar -xf zeromq-4.3.3.tar.gz # cd zeromq-4.3.3 ./configure make make DESTDIR=/path/for/the/library install LD_LIBRARY_PATH=/path/for/the/library/lib R -e \"install.packages('clustermq')\""},{"path":"/articles/userguide.html","id":"r-package","dir":"Articles","previous_headings":"Installation","what":"R package","title":"User Guide","text":"latest stable version available CRAN. Alternatively, also available Github. develop branch, introduce code changes new features. may contain bugs, poor documentation, inconveniences. branch may install times. However, feedback welcome.","code":"# from CRAN install.packages('clustermq') # from Github # install.packages('remotes') remotes::install_github('mschubert/clustermq') # install.packages('remotes') remotes::install_github('mschubert/clustermq', ref=\"develop\")"},{"path":"/articles/userguide.html","id":"configuration","dir":"Articles","previous_headings":"","what":"Configuration","title":"User Guide","text":"Choose preferred parallelism using: three kinds schedulers: LOCAL - sequential processing calls (default HPC scheduler found) Multiple cores machine HPC schedulers Parallel HPC schedulers can also used via SSH.","code":"options(clustermq.scheduler = \"your scheduler here\")"},{"path":"/articles/userguide.html","id":"local-parallelization","dir":"Articles","previous_headings":"Configuration","what":"Local parallelization","title":"User Guide","text":"main focus package, can use parallelize function calls locally multiple cores processes. can also useful test code submitting scheduler. Multiprocess (recommended) - Use callr package run manage multiple parallel R processes options(clustermq.scheduler=\"multiprocess\") Multicore - Uses parallel package fork current R process multiple threads options(clustermq.scheduler=\"multicore\"). sometimes causes problems (macOS, RStudio) available Windows.","code":""},{"path":"/articles/userguide.html","id":"setting-up-the-scheduler","dir":"Articles","previous_headings":"Configuration","what":"Setting up the scheduler","title":"User Guide","text":"HPC cluster’s scheduler ensures computing jobs distributed available worker nodes. Hence, clustermq interfaces order computations. default, take whichever scheduler find fall back local processing. work , cases. set scheduler explicitly, see following links: LSF - work without setup SGE - work without setup SLURM - work without setup PBS/Torque - needs options(clustermq.scheduler=\"PBS\"/\"Torque\") want another scheduler, open issue Default submission templates provided can customized, e.g. activate compute environments containers.","code":""},{"path":"/articles/userguide.html","id":"ssh-connector","dir":"Articles","previous_headings":"Configuration","what":"SSH connector","title":"User Guide","text":"reasons might prefer work computing cluster directly rather local machine instead. RStudio excellent local IDE, ’s responsive feature-rich browser-based solutions (RStudio server, Project Jupyter), avoids X forwarding issues want look plots just made. Using setup, however, lost access computing cluster. Instead, copy data , submit individual scripts jobs, aggregating data end . clustermq trying solve providing transparent SSH interface. order use clustermq local machine, package needs installed computing cluster. computing cluster, set scheduler make sure clustermq runs without problems. Note remote scheduler can LOCAL (default HPC scheduler found) SSH work. local machine, add following options ~/.Rprofile: recommend set SSH keys password-less login.","code":"# If this is set to 'LOCAL' or 'SSH' you will get the following error: # Expected PROXY_READY, received ‘PROXY_ERROR: Remote SSH QSys is not allowed’ options( clustermq.scheduler = \"multiprocess\" # or multicore, LSF, SGE, Slurm etc. ) options( clustermq.scheduler = \"ssh\", clustermq.ssh.host = \"user@host\", # use your user and host, obviously clustermq.ssh.log = \"~/cmq_ssh.log\" # log for easier debugging )"},{"path":[]},{"path":"/articles/userguide.html","id":"the-q-function","dir":"Articles","previous_headings":"Usage","what":"The Q function","title":"User Guide","text":"following arguments supported Q: fun - function call. needs self-sufficient (access master environment) ... - iterated arguments passed function. one, need named const - named list non-iterated arguments passed fun export - named list objects export worker environment Behavior can fine-tuned using options : fail_on_error - Whether stop one calls returns error seed - common seed combined job number reproducible results memory - Amount memory request job (bsub -M) n_jobs - Number jobs submit function calls job_size - Number function calls per job. used combination n_jobs latter overall limit chunk_size - many calls worker process reporting back master. Default: every worker report back 100 times total full documentation available typing ?Q.","code":""},{"path":"/articles/userguide.html","id":"examples","dir":"Articles","previous_headings":"Usage","what":"Examples","title":"User Guide","text":"package designed distribute arbitrary function calls HPC worker nodes. , however, couple caveats observe R session running worker share local memory. simplest example function call completely self-sufficient, one argument (x) iterate : Non-iterated arguments supported const argument: function relies objects environment passed arguments (including functions), can exported using export argument: want use package function need load worker using pkg argument referencing package_name:::","code":"fx = function(x) x * 2 Q(fx, x=1:3, n_jobs=1) #> Running sequentially ('LOCAL') ... #> [[1]] #> [1] 2 #> #> [[2]] #> [1] 4 #> #> [[3]] #> [1] 6 fx = function(x, y) x * 2 + y Q(fx, x=1:3, const=list(y=10), n_jobs=1) #> Running sequentially ('LOCAL') ... #> [[1]] #> [1] 12 #> #> [[2]] #> [1] 14 #> #> [[3]] #> [1] 16 fx = function(x) x * 2 + y Q(fx, x=1:3, export=list(y=10), n_jobs=1) #> Running sequentially ('LOCAL') ... #> [[1]] #> [1] 12 #> #> [[2]] #> [1] 14 #> #> [[3]] #> [1] 16 fx = function(x) { x %>% mutate(area = Sepal.Length * Sepal.Width) %>% head() } Q(fx, x=list(iris), pkgs=\"dplyr\", n_jobs=1) #> Running sequentially ('LOCAL') ... #> #> Attaching package: 'dplyr' #> The following objects are masked from 'package:stats': #> #> filter, lag #> The following objects are masked from 'package:base': #> #> intersect, setdiff, setequal, union #> [[1]] #> Sepal.Length Sepal.Width Petal.Length Petal.Width Species area #> 1 5.1 3.5 1.4 0.2 setosa 17.85 #> 2 4.9 3.0 1.4 0.2 setosa 14.70 #> 3 4.7 3.2 1.3 0.2 setosa 15.04 #> 4 4.6 3.1 1.5 0.2 setosa 14.26 #> 5 5.0 3.6 1.4 0.2 setosa 18.00 #> 6 5.4 3.9 1.7 0.4 setosa 21.06"},{"path":"/articles/userguide.html","id":"as-parallel-foreach-backend","dir":"Articles","previous_headings":"Usage","what":"As parallel foreach backend","title":"User Guide","text":"foreach package provides interface perform repeated tasks different backends. can perform function simple loops using %%: can also perform operations parallel using %dopar%: latter allows registering different handlers parallel execution, can use clustermq: BiocParallel supports foreach , means can run packages use BiocParallel cluster well via DoparParam.","code":"library(foreach) x = foreach(i=1:3) %do% sqrt(i) x = foreach(i=1:3) %dopar% sqrt(i) #> Warning: executing %dopar% sequentially: no parallel backend registered # set up the scheduler first, otherwise this will run sequentially clustermq::register_dopar_cmq(n_jobs=2, memory=1024) # this accepts same arguments as `Q` x = foreach(i=1:3) %dopar% sqrt(i) # this will be executed as jobs #> Running sequentially ('LOCAL') ... library(BiocParallel) register(DoparParam()) # after register_dopar_cmq(...) bplapply(1:3, sqrt)"},{"path":"/articles/userguide.html","id":"with-drake","dir":"Articles","previous_headings":"Usage","what":"With drake","title":"User Guide","text":"drake package enables users define dependency structure different function calls, evaluate underlying data changed. drake — , Data Frames R Make — general-purpose workflow manager data-driven tasks. rebuilds intermediate data objects dependencies change, skips work results already date. every runthrough starts scratch, completed workflows tangible evidence reproducibility. drake also supports scalability, parallel computing, smooth user experience comes setting , deploying, maintaining data science projects. can use clustermq perform calculations jobs:","code":"library(drake) load_mtcars_example() # clean(destroy = TRUE) # options(clustermq.scheduler = \"multicore\") make(my_plan, parallelism = \"clustermq\", jobs = 2, verbose = 4)"},{"path":"/articles/userguide.html","id":"options","dir":"Articles","previous_headings":"","what":"Options","title":"User Guide","text":"various configurable options mentioned throughout documentation, applicable, however, list options reference. Options can set including call options( = ) .Rprofile, calling options( = ) script interactively session. clustermq.scheduler - One supported clustermq schedulers; options \"LOCAL\", \"multiprocess\", \"multicore\", \"lsf\", \"sge\", \"slurm\", \"pbs\", \"Torque\", \"ssh\" (default HPC scheduler found $PATH, otherwise \"LOCAL\") clustermq.host - name node device constructing ZeroMQ host address (default Sys.info()[\"nodename\"]) clustermq.ssh.host - user name host connecting HPC via SSH (e.g. user@host); recommend setting SSH keys password-less login clustermq.ssh.log - Path file (SSH host) created populated logging information regarding SSH connection (e.g. \"~/cmq_ssh.log\"); helpful debugging purposes clustermq.ssh.timeout - amount time wait (seconds) SSH start-connection timing (default 5 seconds) clustermq.worker.timeout - amount time wait (seconds) master-worker communication timing (default 600 seconds) clustermq.error.timeout - amount time wait (seconds), case worker error, remaining workers finish computations shut cleanly (default min(timeout, 30) seconds) clustermq.template - Path template file submitting HPC jobs; necessary using template, otherwise default template used (default depends clustermq.scheduler) clustermq.data.warning - threshold size common data (Mb) clustermq throws warning (default 1000) clustermq.defaults - named-list default values HPC template; takes precedence defaults specified template file (default empty list (.e. list()))","code":""},{"path":[]},{"path":"/articles/userguide.html","id":"debugging-workers","dir":"Articles","previous_headings":"Troubleshooting","what":"Debugging workers","title":"User Guide","text":"Function calls evaluated workers wrapped event handlers, means even call evaluation throws error, reported back main R session. However, reasons workers might crash, case can report back. include: segfault low-level process Process kill due resource constraints (e.g. walltime) Reaching wait timeout without signal master process Probably others case, useful worker(s) create log file also include events reported back. can requested using: create file called -.log current working directory, irrespective scheduler use. can customize file name using Note case log_file template field scheduler script, hence needs present order work. default templates field included. order log worker separately, schedulers support wildcards log file names. instance: Multicore/Multiprocess: log_file=\"/path/.file.%\" SGE: log_file=\"/path/.file.\\$TASK_ID\" LSF: log_file=\"/path/.file.%\" Slurm: log_file=\"/path/.file.%\" PBS: log_file=\"/path/.file.$PBS_ARRAY_INDEX\" Torque: log_file=\"/path/.file.$PBS_ARRAYID\" scheduler documentation details available options. reporting bug includes worker crashes, please always include log file.","code":"Q(..., log_worker=TRUE) Q(..., template=list(log_file = ))"},{"path":"/articles/userguide.html","id":"ssh","dir":"Articles","previous_headings":"Troubleshooting","what":"SSH","title":"User Guide","text":"trying remote schedulers via SSH, make sure scheduler works first connect cluster run job . terminal stuck make sure step SSH connection works typing following commands local terminal make sure don’t get errors warnings step: get Command found: R error, make sure $PATH set correctly ~/.bash_profile /~/.bashrc (depending cluster config might need either). may also need modify SSH template load R module conda environment. get SSH warning error try ssh -v enable verbose output. forward works, run following local R session (ideally also command-line R, RStudio): create log file remote server contain errors might occurred ssh_proxy startup. ssh_proxy startup fails local machine error server log show errors, can try increasing timeout: can happen SSH startup template includes additional steps starting R, activating module conda environment, confirm connection via two-factor authentication.","code":"Connecting via SSH ... # test your ssh login that you set up in ~/.ssh/config # if this fails you have not set up SSH correctly ssh # test port forwarding from 54709 remote to 6687 local (ports are random) # if the fails you will not be able to use clustermq via SSH ssh -R 54709:localhost:6687 R --vanilla options(clustermq.scheduler = \"ssh\", clustermq.ssh.log = \"~/ssh_proxy.log\") Q(identity, x=1, n_jobs=1) Remote R process did not respond after 5 seconds. Check your SSH server log. options(clustermq.ssh.timeout = 30) # in seconds"},{"path":[]},{"path":"/articles/userguide.html","id":"environments-for-workers","dir":"Articles","previous_headings":"Environments","what":"Environments for workers","title":"User Guide","text":"cases, may necessary activate specific computing environment scheduler jobs prior starting worker. can , instance, R installed specific environment container. Examples environments containers : Bash module environments Conda environments Docker/Singularity containers possible activate job submission script (.e., template file). widely untested, look following LSF scheduler (analogous others): template still needs filled, example need pass either set via .Rprofile option:","code":"#BSUB-J {{ job_name }}[1-{{ n_jobs }}] # name of the job / array jobs #BSUB-o {{ log_file | /dev/null }} # stdout + stderr #BSUB-M {{ memory | 4096 }} # Memory requirements in Mbytes #BSUB-R rusage[mem={{ memory | 4096 }}] # Memory requirements in Mbytes ##BSUB-q default # name of the queue (uncomment) ##BSUB-W {{ walltime | 6:00 }} # walltime (uncomment) module load {{ bashenv | default_bash_env }} # or: source activate {{ conda | default_conda_env_name }} # or: your environment activation command ulimit -v $(( 1024 * {{ memory | 4096 }} )) CMQ_AUTH={{ auth }} R --no-save --no-restore -e 'clustermq:::worker(\"{{ master }}\")' Q(..., template=list(bashenv=\"my environment name\")) options( clustermq.defaults = list(bashenv=\"my default env\") )"},{"path":"/articles/userguide.html","id":"running-master-inside-containers","dir":"Articles","previous_headings":"Environments","what":"Running master inside containers","title":"User Guide","text":"master process inside container, accessing HPC scheduler difficult. Containers, including singularity docker, isolate processes inside container host. R process able submit job scheduler found. Note HPC node running master process must allowed submit jobs. HPC systems allow compute nodes submit jobs. case, may need run master process login node, discuss issue system administrator. container binary compatible host, may able bind scheduler executable container. example, PBS might look something like: working example binding SLURM CentOS 7 container image CentOS 7 host available https://groups.google.com//lbl.gov/d/msg/singularity/syLcsIWWzdo/NZvF2Ud2AAAJ Alternatively, can create script uses SSH execute scheduler login node. , need SSH client container, keys set password-less login, create script call scheduler login node via ssh (e.g. ~/bin/qsub SGE/PBS/Torque, bsub LSF sbatch Slurm): Make sure script executable, bind/copy container somewhere $PATH. Home directories bound default singularity.","code":"#PBS directives ... module load singularity SINGULARITYENV_APPEND_PATH=/opt/pbs/bin singularity exec --bind /opt/pbs/bin r_image.sif Rscript master_script.R #!/bin/bash ssh -i ~/.ssh/ ${PBS_O_HOST:-\"no_host_not_in_a_pbs_job\"} qsub \"$@\" chmod u+x ~/bin/qsub SINGULARITYENV_APPEND_PATH=~/bin"},{"path":[]},{"path":"/articles/userguide.html","id":"LSF","dir":"Articles","previous_headings":"Scheduler templates","what":"LSF","title":"User Guide","text":"~/.Rprofile computing cluster, set following options: option clustermq.template point LSF template file like one (needed want supply template rather using default). file, #BSUB-* defines command-line arguments bsub program. Memory: defined BSUB-M BSUB-R. Check local setup memory values supplied MiB KiB, default 4096 requesting memory calling Q() Queue: BSUB-q default. Use queue name default. likely exist system, choose right name (comment line additional #) Walltime: BSUB-W {{ walltime }}. Set maximum time job allowed run killed. default disable line. enable , enter fixed value pass walltime argument function call. way written, use 6 hours arguemnt given. options, see LSF documentation add via #BSUB-* (* represents argument) change identifiers curly braces ({{ ... }}), used fill right variables done, package use settings longer warn missing options.","code":"options( clustermq.scheduler = \"lsf\", clustermq.template = \"/path/to/file/below\" # if using your own template ) #BSUB-J {{ job_name }}[1-{{ n_jobs }}] # name of the job / array jobs #BSUB-n {{ cores | 1 }} # number of cores to use per job #BSUB-o {{ log_file | /dev/null }} # stdout + stderr; %I for array index #BSUB-M {{ memory | 4096 }} # Memory requirements in Mbytes #BSUB-R rusage[mem={{ memory | 4096 }}] # Memory requirements in Mbytes ##BSUB-q default # name of the queue (uncomment) ##BSUB-W {{ walltime | 6:00 }} # walltime (uncomment) ulimit -v $(( 1024 * {{ memory | 4096 }} )) CMQ_AUTH={{ auth }} R --no-save --no-restore -e 'clustermq:::worker(\"{{ master }}\")'"},{"path":"/articles/userguide.html","id":"SGE","dir":"Articles","previous_headings":"Scheduler templates","what":"SGE","title":"User Guide","text":"~/.Rprofile computing cluster, set following options: option clustermq.template point SGE template file like one (needed want supply template rather using default). file, #$-* defines command-line arguments qsub program. Queue: $ -q default. Use queue name default. likely exist system, choose right name (comment line additional #) options, see SGE documentation. change identifiers curly braces ({{ ... }}), used fill right variables. done, package use settings longer warn missing options.","code":"options( clustermq.scheduler = \"sge\", clustermq.template = \"/path/to/file/below\" # if using your own template ) #$ -N {{ job_name }} # job name #$ -q default # submit to queue named \"default\" #$ -j y # combine stdout/error in one file #$ -o {{ log_file | /dev/null }} # output file #$ -cwd # use pwd as work dir #$ -V # use environment variable #$ -t 1-{{ n_jobs }} # submit jobs as array #$ -pe {{ cores | 1 }} # number of cores to use per job ulimit -v $(( 1024 * {{ memory | 4096 }} )) CMQ_AUTH={{ auth }} R --no-save --no-restore -e 'clustermq:::worker(\"{{ master }}\")'"},{"path":"/articles/userguide.html","id":"SLURM","dir":"Articles","previous_headings":"Scheduler templates","what":"SLURM","title":"User Guide","text":"~/.Rprofile computing cluster, set following options: option clustermq.template point SLURM template file like one (needed want supply template rather using default). file, #SBATCH defines command-line arguments sbatch program. Queue: SBATCH --partition default. Use queue name default. likely exist system, choose right name (comment line additional #) options, see SLURM documentation. change identifiers curly braces ({{ ... }}), used fill right variables. done, package use settings longer warn missing options.","code":"options( clustermq.scheduler = \"slurm\", clustermq.template = \"/path/to/file/below\" # if using your own template ) #!/bin/sh #SBATCH --job-name={{ job_name }} #SBATCH --partition=default #SBATCH --output={{ log_file | /dev/null }} # you can add .%a for array index #SBATCH --error={{ log_file | /dev/null }} #SBATCH --mem-per-cpu={{ memory | 4096 }} #SBATCH --array=1-{{ n_jobs }} #SBATCH --cpus-per-task={{ cores | 1 }} ulimit -v $(( 1024 * {{ memory | 4096 }} )) CMQ_AUTH={{ auth }} R --no-save --no-restore -e 'clustermq:::worker(\"{{ master }}\")'"},{"path":"/articles/userguide.html","id":"PBS","dir":"Articles","previous_headings":"Scheduler templates","what":"PBS","title":"User Guide","text":"~/.Rprofile computing cluster, set following options: option clustermq.template point PBS template file like one (needed want supply template rather using default). file, #PBS-* defines command-line arguments qsub program. Queue: #PBS-q default. Use queue name default. likely exist system, choose right name (comment line additional #) options, see PBS documentation. change identifiers curly braces ({{ ... }}), used fill right variables. done, package use settings longer warn missing options.","code":"options( clustermq.scheduler = \"pbs\", clustermq.template = \"/path/to/file/below\" # if using your own template ) #PBS -N {{ job_name }} #PBS -J 1-{{ n_jobs }} #PBS -l select=1:ncpus={{ cores | 1 }}:mpiprocs={{ cores | 1 }}:mem={{ memory | 4096 }}MB #PBS -l walltime={{ walltime | 12:00:00 }} #PBS -o {{ log_file | /dev/null }} #PBS -j oe #PBS -q default ulimit -v $(( 1024 * {{ memory | 4096 }} )) CMQ_AUTH={{ auth }} R --no-save --no-restore -e 'clustermq:::worker(\"{{ master }}\")'"},{"path":"/articles/userguide.html","id":"Torque","dir":"Articles","previous_headings":"Scheduler templates","what":"Torque","title":"User Guide","text":"~/.Rprofile computing cluster, set following options: option clustermq.template point Torque template file like one (needed want supply template rather using default). file, #PBS-* defines command-line arguments qsub program. Queue: #PBS -q default. Use queue name default. likely exist system, choose right name (comment line additional #) options, see Torque documentation. change identifiers curly braces ({{ ... }}), used fill right variables. done, package use settings longer warn missing options.","code":"options(clustermq.scheduler = \"Torque\", clustermq.template = \"/path/to/file/below\" # if using your own template ) #PBS -N {{ job_name }} #PBS -l nodes={{ n_jobs }}:ppn={{ cores | 1 }},walltime={{ walltime | 12:00:00 }} #PBS -o {{ log_file | /dev/null }} #PBS -q default #PBS -j oe ulimit -v $(( 1024 * {{ memory | 4096 }} )) CMQ_AUTH={{ auth }} R --no-save --no-restore -e 'clustermq:::worker(\"{{ master }}\")'"},{"path":"/articles/userguide.html","id":"ssh-template","dir":"Articles","previous_headings":"Scheduler templates","what":"SSH","title":"User Guide","text":"SSH scheduler, can access remote schedulers via SSH. want use , first make sure real scheduler running manually connecting HPC environment. default template shown . R HPC $PATH, may need specify path load required bash modules/conda environments.","code":"options(clustermq.scheduler = \"ssh\", clustermq.ssh.host = \"myhost\", # set this up in your local ~/.ssh/config clustermq.ssh.log = \"~/ssh_proxy.log\", # log file on your HPC clustermq.ssh.timeout = 30, # if changing the default connection timeout clustermq.template = \"/path/to/file/below\" # if using your own template ) ssh -o \"ExitOnForwardFailure yes\" -f \\ -R {{ ctl_port }}:localhost:{{ local_port }} \\ -R {{ job_port }}:localhost:{{ fwd_port }} \\ {{ ssh_host }} \\ \"R --no-save --no-restore -e \\ 'clustermq:::ssh_proxy(ctl={{ ctl_port }}, job={{ job_port }})' \\ > {{ ssh_log | /dev/null }} 2>&1\""},{"path":"/authors.html","id":null,"dir":"","previous_headings":"","what":"Authors","title":"Authors and Citation","text":"Michael Schubert. Author, maintainer.","code":""},{"path":"/authors.html","id":"citation","dir":"","previous_headings":"","what":"Citation","title":"Authors and Citation","text":"Schubert, M. clustermq enables efficient parallelisation genomic analyses. Bioinformatics (2019). doi:10.1093/bioinformatics/btz284","code":"@Article{, title = {clustermq enables efficient parallelisation of genomic analyses}, author = {Michael Schubert}, journal = {Bioinformatics}, month = {May}, year = {2019}, language = {en}, doi = {10.1093/bioinformatics/btz284}, url = {https://github.com/mschubert/clustermq}, }"},{"path":"/index.html","id":"clustermq-send-r-function-calls-as-cluster-jobs","dir":"","previous_headings":"","what":"Evaluate Function Calls on HPC Schedulers (LSF, SGE, SLURM, PBS/Torque)","title":"Evaluate Function Calls on HPC Schedulers (LSF, SGE, SLURM, PBS/Torque)","text":"package allow send function calls jobs computing cluster minimal interface provided Q function: Computations done entirely network without temporary files network-mounted storage, strain file system apart starting R per job. calculations load-balanced, .e. workers get jobs done faster also receive function calls work . especially useful calls return time, one worker high load. Browse vignettes : User Guide Technical Documentation","code":"# load the library and create a simple function library(clustermq) fx = function(x) x * 2 # queue the function call on your scheduler Q(fx, x=1:3, n_jobs=1) # list(2,4,6)"},{"path":"/index.html","id":"installation","dir":"","previous_headings":"","what":"Installation","title":"Evaluate Function Calls on HPC Schedulers (LSF, SGE, SLURM, PBS/Torque)","text":"First, need ZeroMQ system library. probably already installed system. , package manager provide : install clustermq package R CRAN: Alternatively can use remotes package install directly Github:","code":"# You can skip this step on Windows and macOS, the package binary has it # On a computing cluster, we recommend to use Conda or Linuxbrew brew install zeromq # Linuxbrew, Homebrew on macOS conda install zeromq # Conda, Miniconda sudo apt-get install libzmq3-dev # Ubuntu sudo yum install zeromq-devel # Fedora pacman -S zeromq # Arch Linux install.packages('clustermq') # install.packages('remotes') remotes::install_github('mschubert/clustermq') # remotes::install_github('mschubert/clustermq', ref=\"develop\") # dev version"},{"path":"/index.html","id":"schedulers","dir":"","previous_headings":"","what":"Schedulers","title":"Evaluate Function Calls on HPC Schedulers (LSF, SGE, SLURM, PBS/Torque)","text":"HPC cluster’s scheduler ensures computing jobs distributed available worker nodes. Hence, clustermq interfaces order computations. currently support following schedulers (either locally via SSH): Multiprocess - test calls parallelize cores using options(clustermq.scheduler=\"multiprocess\") LSF - work without setup SGE - work without setup SLURM - work without setup PBS/Torque - needs options(clustermq.scheduler=\"PBS\"/\"Torque\") via SSH - needs options(clustermq.scheduler=\"ssh\", clustermq.ssh.host=) Default submission templates provided can customized, e.g. activate compute environments containers.","code":""},{"path":"/index.html","id":"usage","dir":"","previous_headings":"","what":"Usage","title":"Evaluate Function Calls on HPC Schedulers (LSF, SGE, SLURM, PBS/Torque)","text":"common arguments Q : fun - function call. needs self-sufficient (access master environment) ... - iterated arguments passed function. one, need named const - named list non-iterated arguments passed fun export - named list objects export worker environment documentation arguments can accessed typing ?Q. Examples using const export : clustermq can also used parallel backend foreach. also used BiocParallel, can run packages cluster well: examples available user guide.","code":"# adding a constant argument fx = function(x, y) x * 2 + y Q(fx, x=1:3, const=list(y=10), n_jobs=1) # exporting an object to workers fx = function(x) x * 2 + y Q(fx, x=1:3, export=list(y=10), n_jobs=1) library(foreach) register_dopar_cmq(n_jobs=2, memory=1024) # see `?workers` for arguments foreach(i=1:3) %dopar% sqrt(i) # this will be executed as jobs library(BiocParallel) register(DoparParam()) # after register_dopar_cmq(...) bplapply(1:3, sqrt)"},{"path":"/index.html","id":"comparison-to-other-packages","dir":"","previous_headings":"","what":"Comparison to other packages","title":"Evaluate Function Calls on HPC Schedulers (LSF, SGE, SLURM, PBS/Torque)","text":"packages provide high-level parallelization R function calls computing cluster. compared clustermq BatchJobs batchtools processing many short-running jobs, found approximately 1000x less overhead cost. Overhead comparison short, use clustermq want: one-line solution run cluster jobs minimal setup access cluster functions local Rstudio via SSH fast processing many function calls without network storage /O Use batchtools : want use mature well-tested package don’t mind arguments every call written /read disc don’t mind ’s load-balancing run-time Use Snakemake drake : want design run workflow HPC Don’t use batch (last updated 2013) BatchJobs (issues SQLite network-mounted storage).","code":""},{"path":"/index.html","id":"contributing","dir":"","previous_headings":"","what":"Contributing","title":"Evaluate Function Calls on HPC Schedulers (LSF, SGE, SLURM, PBS/Torque)","text":"use Github’s Issue Tracker coordinate development clustermq. Contributions welcome come many different forms, shapes, sizes. include, limited : Questions: welcome ask questions something clear User guide. Bug reports: Let us know something work expected. sure include self-contained Minimal Reproducible Example set log_worker=TRUE. Code contributions: look good first issue tag. Please discuss anything complicated putting lot work , ’m happy help get started.","code":""},{"path":"/index.html","id":"citation","dir":"","previous_headings":"","what":"Citation","title":"Evaluate Function Calls on HPC Schedulers (LSF, SGE, SLURM, PBS/Torque)","text":"project part academic work, evaluated citations. like able continue working research support tools like clustermq, please cite article using publications: M Schubert. clustermq enables efficient parallelisation genomic analyses. Bioinformatics (2019). doi:10.1093/bioinformatics/btz284","code":""},{"path":"/reference/LOCAL.html","id":null,"dir":"Reference","previous_headings":"","what":"Placeholder for local processing — LOCAL","title":"Placeholder for local processing — LOCAL","text":"Mainly tests pass without setting scheduler","code":""},{"path":"/reference/LSF.html","id":null,"dir":"Reference","previous_headings":"","what":"LSF scheduler functions — LSF","title":"LSF scheduler functions — LSF","text":"Derives QSys provide LSF-specific functions","code":""},{"path":"/reference/MULTICORE.html","id":null,"dir":"Reference","previous_headings":"","what":"Process on multiple cores on one machine — MULTICORE","title":"Process on multiple cores on one machine — MULTICORE","text":"Derives QSys provide multicore-specific functions","code":""},{"path":"/reference/MULTIPROCESS.html","id":null,"dir":"Reference","previous_headings":"","what":"Process on multiple processes on one machine — MULTIPROCESS","title":"Process on multiple processes on one machine — MULTIPROCESS","text":"Derives QSys provide callr-specific functions","code":""},{"path":"/reference/Q.html","id":null,"dir":"Reference","previous_headings":"","what":"Queue function calls on the cluster — Q","title":"Queue function calls on the cluster — Q","text":"Queue function calls cluster","code":""},{"path":"/reference/Q.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Queue function calls on the cluster — Q","text":"","code":"Q( fun, ..., const = list(), export = list(), pkgs = c(), seed = 128965, memory = NULL, template = list(), n_jobs = NULL, job_size = NULL, split_array_by = -1, rettype = \"list\", fail_on_error = TRUE, workers = NULL, log_worker = FALSE, chunk_size = NA, timeout = Inf, max_calls_worker = Inf, verbose = TRUE )"},{"path":"/reference/Q.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Queue function calls on the cluster — Q","text":"fun function call ... Objects iterated function call const list constant arguments passed function call export List objects exported worker pkgs Character vector packages load worker seed seed set function call memory Short template=list(memory=value) template named list values fill template n_jobs number LSF jobs submit; upper limit jobs job_size given well job_size number function calls per job split_array_by dimension number split arrays `...`; default: last rettype Return type function call (vector type 'list') fail_on_error error occurs workers, continue fail? workers Optional instance QSys representing worker pool log_worker Write log file worker chunk_size Number function calls chunk together defaults 100 chunks per worker max. 10 kb per chunk timeout Maximum time seconds wait worker (default: Inf) max_calls_worker Maxmimum number function calls sent one worker verbose Print status messages progress bar (default: TRUE)","code":""},{"path":"/reference/Q.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Queue function calls on the cluster — Q","text":"list whatever `fun` returned","code":""},{"path":"/reference/Q.html","id":"ref-examples","dir":"Reference","previous_headings":"","what":"Examples","title":"Queue function calls on the cluster — Q","text":"","code":"if (FALSE) { # Run a simple multiplication for numbers 1 to 3 on a worker node fx = function(x) x * 2 Q(fx, x=1:3, n_jobs=1) # list(2,4,6) # Run a mutate() call in dplyr on a worker node iris %>% mutate(area = Q(`*`, e1=Sepal.Length, e2=Sepal.Width, n_jobs=1)) # iris with an additional column 'area' }"},{"path":"/reference/QSys.html","id":null,"dir":"Reference","previous_headings":"","what":"Class for basic queuing system functions — QSys","title":"Class for basic queuing system functions — QSys","text":"Provides basic functions needed communicate machines abstract functions rZMQ scheduler implementations can rely higher level functionality","code":""},{"path":"/reference/Q_rows.html","id":null,"dir":"Reference","previous_headings":"","what":"Queue function calls defined by rows in a data.frame — Q_rows","title":"Queue function calls defined by rows in a data.frame — Q_rows","text":"Queue function calls defined rows data.frame","code":""},{"path":"/reference/Q_rows.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Queue function calls defined by rows in a data.frame — Q_rows","text":"","code":"Q_rows( df, fun, const = list(), export = list(), pkgs = c(), seed = 128965, memory = NULL, template = list(), n_jobs = NULL, job_size = NULL, rettype = \"list\", fail_on_error = TRUE, workers = NULL, log_worker = FALSE, chunk_size = NA, timeout = Inf, max_calls_worker = Inf, verbose = TRUE )"},{"path":"/reference/Q_rows.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Queue function calls defined by rows in a data.frame — Q_rows","text":"df data.frame iterated arguments fun function call const list constant arguments passed function call export List objects exported worker pkgs Character vector packages load worker seed seed set function call memory Short template=list(memory=value) template named list values fill template n_jobs number LSF jobs submit; upper limit jobs job_size given well job_size number function calls per job rettype Return type function call (vector type 'list') fail_on_error error occurs workers, continue fail? workers Optional instance QSys representing worker pool log_worker Write log file worker chunk_size Number function calls chunk together defaults 100 chunks per worker max. 10 kb per chunk timeout Maximum time seconds wait worker (default: Inf) max_calls_worker Maxmimum number function calls sent one worker verbose Print status messages progress bar (default: TRUE)","code":""},{"path":"/reference/Q_rows.html","id":"ref-examples","dir":"Reference","previous_headings":"","what":"Examples","title":"Queue function calls defined by rows in a data.frame — Q_rows","text":"","code":"if (FALSE) { # Run a simple multiplication for data frame columns x and y on a worker node fx = function (x, y) x * y df = data.frame(x = 5, y = 10) Q_rows(df, fx, job_size = 1) # [1] 50 # Q_rows also matches the names of a data frame with the function arguments fx = function (x, y) x - y df = data.frame(y = 5, x = 10) Q_rows(df, fx, job_size = 1) # [1] 5 }"},{"path":"/reference/SGE.html","id":null,"dir":"Reference","previous_headings":"","what":"SGE scheduler functions — SGE","title":"SGE scheduler functions — SGE","text":"Derives QSys provide SGE-specific functions","code":""},{"path":"/reference/SLURM.html","id":null,"dir":"Reference","previous_headings":"","what":"SLURM scheduler functions — SLURM","title":"SLURM scheduler functions — SLURM","text":"Derives QSys provide SLURM-specific functions","code":""},{"path":"/reference/SSH.html","id":null,"dir":"Reference","previous_headings":"","what":"SSH scheduler functions — SSH","title":"SSH scheduler functions — SSH","text":"Derives QSys provide SSH-specific functions","code":""},{"path":"/reference/ZeroMQ.html","id":null,"dir":"Reference","previous_headings":"","what":"Wrap C++ Rcpp module in R6 to get reliable argument matching — ZeroMQ","title":"Wrap C++ Rcpp module in R6 to get reliable argument matching — ZeroMQ","text":"R6 wrapper C++ class order support R argument matching. Ideally, Rcpp point support natively file longer necessary. , causes redundancy zeromq.cpp, small inconvenience much less error-prone relying positional arguments.","code":""},{"path":"/reference/bind_avail.html","id":null,"dir":"Reference","previous_headings":"","what":"Binds a ZeroMQ socket to an available port in given range — bind_avail","title":"Binds a ZeroMQ socket to an available port in given range — bind_avail","text":"Binds ZeroMQ socket available port given range","code":""},{"path":"/reference/bind_avail.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Binds a ZeroMQ socket to an available port in given range — bind_avail","text":"","code":"bind_avail(socket, range, iface = \"tcp://*\", n_tries = 100)"},{"path":"/reference/bind_avail.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Binds a ZeroMQ socket to an available port in given range — bind_avail","text":"socket ZeroMQ socket object range Numbers consider (e.g. 6000:8000) iface Interface listen n_tries Number ports try range","code":""},{"path":"/reference/bind_avail.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Binds a ZeroMQ socket to an available port in given range — bind_avail","text":"port socket bound ","code":""},{"path":"/reference/check_args.html","id":null,"dir":"Reference","previous_headings":"","what":"Function to check arguments with which Q() is called — check_args","title":"Function to check arguments with which Q() is called — check_args","text":"Function check arguments Q() called","code":""},{"path":"/reference/check_args.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Function to check arguments with which Q() is called — check_args","text":"","code":"check_args(fun, iter, const = list())"},{"path":"/reference/check_args.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Function to check arguments with which Q() is called — check_args","text":"fun function call iter Objects iterated function call const list constant arguments passed function call","code":""},{"path":"/reference/check_args.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Function to check arguments with which Q() is called — check_args","text":"Processed iterated argument list 'iter' list","code":""},{"path":"/reference/chunk.html","id":null,"dir":"Reference","previous_headings":"","what":"Subset index chunk for processing — chunk","title":"Subset index chunk for processing — chunk","text":"'attr' `[.data.frame` takes much CPU time","code":""},{"path":"/reference/chunk.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Subset index chunk for processing — chunk","text":"","code":"chunk(x, i)"},{"path":"/reference/chunk.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Subset index chunk for processing — chunk","text":"x Index data.frame Rows subset","code":""},{"path":"/reference/chunk.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Subset index chunk for processing — chunk","text":"x[,]","code":""},{"path":"/reference/clustermq.html","id":null,"dir":"Reference","previous_headings":"","what":"Evaluate Function Calls on HPC Schedulers (LSF, SGE, SLURM) — clustermq","title":"Evaluate Function Calls on HPC Schedulers (LSF, SGE, SLURM) — clustermq","text":"Provides Q function send arbitrary function calls workers HPC schedulers without relying network-mounted storage. Allows using remote schedulers via SSH.","code":""},{"path":"/reference/clustermq.html","id":"details","dir":"Reference","previous_headings":"","what":"Details","title":"Evaluate Function Calls on HPC Schedulers (LSF, SGE, SLURM) — clustermq","text":"hood, submit cluster job connects master via TCP master send function argument chunks worker worker return results master everything done get back result Computations done entirely network without temporary files network-mounted storage, strain file system apart starting R per job. removes biggest bottleneck distributed computing. Using approach, can easily load-balancing, .e. workers get jobs done faster also receive function calls work . especially useful calls return time, one worker high load. detailed usage instructions, see documentation Q function.","code":""},{"path":"/reference/cmq_foreach.html","id":null,"dir":"Reference","previous_headings":"","what":"clustermq foreach handler — cmq_foreach","title":"clustermq foreach handler — cmq_foreach","text":"clustermq foreach handler","code":""},{"path":"/reference/cmq_foreach.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"clustermq foreach handler — cmq_foreach","text":"","code":"cmq_foreach(obj, expr, envir, data)"},{"path":"/reference/cmq_foreach.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"clustermq foreach handler — cmq_foreach","text":"obj Returned foreach::foreach, containing following variables: args : Arguments passed, call argnames: character vector arguments passed evalenv : Environment evaluate arguments export : character vector variable names export nodes packages: character vector required packages verbose : whether print status messages [logical] errorHandling: string function name call error , e.g. \"stop\" expr R expression curly braces envir Environment evaluate arguments data Common arguments passed register_dopcar_cmq(), e.g. n_jobs","code":""},{"path":"/reference/dot-onAttach.html","id":null,"dir":"Reference","previous_headings":"","what":"Report queueing system on package attach if not set — .onAttach","title":"Report queueing system on package attach if not set — .onAttach","text":"Report queueing system package attach set","code":""},{"path":"/reference/dot-onAttach.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Report queueing system on package attach if not set — .onAttach","text":"","code":".onAttach(libname, pkgname)"},{"path":"/reference/dot-onAttach.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Report queueing system on package attach if not set — .onAttach","text":"libname default arg compatibility pkgname default arg compatibility","code":""},{"path":"/reference/dot-onLoad.html","id":null,"dir":"Reference","previous_headings":"","what":"Select the queueing system on package loading — .onLoad","title":"Select the queueing system on package loading — .onLoad","text":"done setting variable 'qsys' package environment object contains desired queueing system.","code":""},{"path":"/reference/dot-onLoad.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Select the queueing system on package loading — .onLoad","text":"","code":".onLoad(libname, pkgname)"},{"path":"/reference/dot-onLoad.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Select the queueing system on package loading — .onLoad","text":"libname default arg compatibility pkgname default arg compatibility","code":""},{"path":"/reference/fill_template.html","id":null,"dir":"Reference","previous_headings":"","what":"Fill a template string with supplied values — fill_template","title":"Fill a template string with supplied values — fill_template","text":"Fill template string supplied values","code":""},{"path":"/reference/fill_template.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Fill a template string with supplied values — fill_template","text":"","code":"fill_template(template, values, required = c())"},{"path":"/reference/fill_template.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Fill a template string with supplied values — fill_template","text":"template character string submission template values named list key-value pairs required Keys must present template (default: none)","code":""},{"path":"/reference/fill_template.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Fill a template string with supplied values — fill_template","text":"template placeholder fields replaced values","code":""},{"path":"/reference/host.html","id":null,"dir":"Reference","previous_headings":"","what":"Construct the ZeroMQ host address — host","title":"Construct the ZeroMQ host address — host","text":"Construct ZeroMQ host address","code":""},{"path":"/reference/host.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Construct the ZeroMQ host address — host","text":"","code":"host( node = getOption(\"clustermq.host\", Sys.info()[\"nodename\"]), ports = 6000:9999, n = 100 )"},{"path":"/reference/host.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Construct the ZeroMQ host address — host","text":"node Node device name ports Range ports consider n many addresses return short Whether use unqualified host name (first dot)","code":""},{"path":"/reference/host.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Construct the ZeroMQ host address — host","text":"possible addresses character vector","code":""},{"path":"/reference/master.html","id":null,"dir":"Reference","previous_headings":"","what":"Master controlling the workers — master","title":"Master controlling the workers — master","text":"exchanging messages master workers works following way: * submitted job know start * starts, sends message list(id=0) indicating ready * send function definition common data * also send first data set work * get id > 0, result store * send next data set/index work * computatons complete, send id=0 worker * responds id=-1 (usage stats) shuts ","code":""},{"path":"/reference/master.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Master controlling the workers — master","text":"","code":"master( qsys, iter, rettype = \"list\", fail_on_error = TRUE, chunk_size = NA, timeout = Inf, max_calls_worker = Inf, verbose = TRUE )"},{"path":"/reference/master.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Master controlling the workers — master","text":"qsys Instance QSys object iter Objects iterated function call rettype Return type function fail_on_error error occurs workers, continue fail? chunk_size Number function calls chunk together defaults 100 chunks per worker max. 500 kb per chunk timeout Maximum time seconds wait worker (default: Inf) max_calls_worker Maxmimum number function calls sent one worker verbose Print progress messages","code":""},{"path":"/reference/master.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Master controlling the workers — master","text":"list whatever `fun` returned","code":""},{"path":"/reference/purrr_lookup.html","id":null,"dir":"Reference","previous_headings":"","what":"Lookup table for return types to purrr functions — purrr_lookup","title":"Lookup table for return types to purrr functions — purrr_lookup","text":"Lookup table return types purrr functions","code":""},{"path":"/reference/purrr_lookup.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Lookup table for return types to purrr functions — purrr_lookup","text":"","code":"purrr_lookup"},{"path":"/reference/purrr_lookup.html","id":"format","dir":"Reference","previous_headings":"","what":"Format","title":"Lookup table for return types to purrr functions — purrr_lookup","text":"object class list length 9.","code":""},{"path":"/reference/register_dopar_cmq.html","id":null,"dir":"Reference","previous_headings":"","what":"Register clustermq as `foreach` parallel handler — register_dopar_cmq","title":"Register clustermq as `foreach` parallel handler — register_dopar_cmq","text":"Register clustermq `foreach` parallel handler","code":""},{"path":"/reference/register_dopar_cmq.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Register clustermq as `foreach` parallel handler — register_dopar_cmq","text":"","code":"register_dopar_cmq(...)"},{"path":"/reference/register_dopar_cmq.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Register clustermq as `foreach` parallel handler — register_dopar_cmq","text":"... List arguments passed `Q` function, e.g. n_jobs","code":""},{"path":"/reference/ssh_proxy.html","id":null,"dir":"Reference","previous_headings":"","what":"SSH proxy for different schedulers — ssh_proxy","title":"SSH proxy for different schedulers — ssh_proxy","text":"call manually, SSH qsys ","code":""},{"path":"/reference/ssh_proxy.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"SSH proxy for different schedulers — ssh_proxy","text":"","code":"ssh_proxy(ctl, job, qsys_id = qsys_default)"},{"path":"/reference/ssh_proxy.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"SSH proxy for different schedulers — ssh_proxy","text":"ctl port connect master proxy control job port connect master job control qsys_id Character string QSys class use","code":""},{"path":"/reference/summarize_result.html","id":null,"dir":"Reference","previous_headings":"","what":"Print a summary of errors and warnings that occurred during processing — summarize_result","title":"Print a summary of errors and warnings that occurred during processing — summarize_result","text":"Print summary errors warnings occurred processing","code":""},{"path":"/reference/summarize_result.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Print a summary of errors and warnings that occurred during processing — summarize_result","text":"","code":"summarize_result( result, n_errors, n_warnings, cond_msgs, at = length(result), fail_on_error = TRUE )"},{"path":"/reference/summarize_result.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Print a summary of errors and warnings that occurred during processing — summarize_result","text":"result list vector processing result n_errors many errors occurred n_warnings many warnings occurred cond_msgs Error warnings messages, display first 50 many calls procesed point fail_on_error Stop error(s) occurred","code":""},{"path":"/reference/vec_lookup.html","id":null,"dir":"Reference","previous_headings":"","what":"Lookup table for return types to vector NAs — vec_lookup","title":"Lookup table for return types to vector NAs — vec_lookup","text":"Lookup table return types vector NAs","code":""},{"path":"/reference/vec_lookup.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Lookup table for return types to vector NAs — vec_lookup","text":"","code":"vec_lookup"},{"path":"/reference/vec_lookup.html","id":"format","dir":"Reference","previous_headings":"","what":"Format","title":"Lookup table for return types to vector NAs — vec_lookup","text":"object class list length 9.","code":""},{"path":"/reference/work_chunk.html","id":null,"dir":"Reference","previous_headings":"","what":"Function to process a chunk of calls — work_chunk","title":"Function to process a chunk of calls — work_chunk","text":"chunk comes encapsulated data.frame","code":""},{"path":"/reference/work_chunk.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Function to process a chunk of calls — work_chunk","text":"","code":"work_chunk( df, fun, const = list(), rettype = \"list\", common_seed = NULL, progress = FALSE )"},{"path":"/reference/work_chunk.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Function to process a chunk of calls — work_chunk","text":"df data.frame call IDs rownames arguments columns fun function call const Constant arguments passed call rettype Return type function common_seed seed offset common function calls progress Logical indicated whether display progress bar","code":""},{"path":"/reference/work_chunk.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Function to process a chunk of calls — work_chunk","text":"list call results (try-error failed)","code":""},{"path":"/reference/worker.html","id":null,"dir":"Reference","previous_headings":"","what":"R worker submitted as cluster job — worker","title":"R worker submitted as cluster job — worker","text":"call manually, master ","code":""},{"path":"/reference/worker.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"R worker submitted as cluster job — worker","text":"","code":"worker( master, ..., verbose = TRUE )"},{"path":"/reference/worker.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"R worker submitted as cluster job — worker","text":"master master address (tcp://ip:port) ... Catch-break older template values (ignored) verbose Whether print debug messages","code":""},{"path":"/reference/workers.html","id":null,"dir":"Reference","previous_headings":"","what":"Creates a pool of workers — workers","title":"Creates a pool of workers — workers","text":"Creates pool workers","code":""},{"path":"/reference/workers.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Creates a pool of workers — workers","text":"","code":"workers( n_jobs, data = NULL, reuse = TRUE, template = list(), log_worker = FALSE, qsys_id = getOption(\"clustermq.scheduler\", qsys_default), verbose = FALSE, ... )"},{"path":"/reference/workers.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Creates a pool of workers — workers","text":"n_jobs Number jobs submit (0 implies local processing) data Set common data (function, constant args, seed) reuse Whether workers reusable get shut call template named list values fill template log_worker Write log file worker qsys_id Character string QSys class use verbose Print message worker startup ... Additional arguments passed qsys constructor","code":""},{"path":"/reference/workers.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Creates a pool of workers — workers","text":"instance QSys class","code":""},{"path":"/news/index.html","id":"clustermq-0895","dir":"Changelog","previous_headings":"","what":"clustermq 0.8.95","title":"clustermq 0.8.95","text":"CRAN release: 2020-07-01 now using ZeroMQ via Rcpp preparation v0.9 (#151) New multiprocess backend via callr instead forking (#142, #197) FIXME: test case Sending data sockets now blocking avoid excess memory usage (#161) multicore, multiprocess schedulers now support logging (#169) New option clustermq.host can specify host IP network interface name (#170) Template filling now raise error missing keys (#174, #198) Workers failing large common data improved (fixed?) (#146, #179, #191) Local connections now routed via 127.0.0.1 instead localhost (#192) Submit messages different local, multicore HPC (#196) Functions exported foreach now environment stripped (#200) Deprecation log_worker=T/F argument rescinded","code":""},{"path":"/news/index.html","id":"clustermq-089","dir":"Changelog","previous_headings":"","what":"clustermq 0.8.9","title":"clustermq 0.8.9","text":"CRAN release: 2020-02-29 New option clustermq.ssh.timeout SSH proxy startup (#157) @brendanf New option clustermq.worker.timeout delay worker shutdown (#188) Fixed PBS/Torque docs, template cleanup (#184, #186) @mstr3336 Warning common data large, set clustermq.data.warning (#189)","code":""},{"path":"/news/index.html","id":"clustermq-088","dir":"Changelog","previous_headings":"","what":"clustermq 0.8.8","title":"clustermq 0.8.8","text":"CRAN release: 2019-06-05 Q, Q_rows new arguments verbose (#111) pkgs (#144) foreach backend now uses dedicated API possible (#143, #144) Number size objects common calls now work properly Templates filled internally longer depend infuser package","code":""},{"path":"/news/index.html","id":"clustermq-087","dir":"Changelog","previous_headings":"","what":"clustermq 0.8.7","title":"clustermq 0.8.7","text":"CRAN release: 2019-04-15 Q now max_calls_worker argument avoid walltime (#110) Submission messages now list size common data (drake#800) default templates now optional cores per job field (#123) foreach now treats .export (#124) .combine (#126) correctly New option clustermq.error.timeout wait clean shutdown (#134) SSH command now specified via template file (#122) SSH now forward errors local process (#135) Wiki deprecated, use https://mschubert.github.io/clustermq/ instead","code":""},{"path":"/news/index.html","id":"clustermq-086","dir":"Changelog","previous_headings":"","what":"clustermq 0.8.6","title":"clustermq 0.8.6","text":"CRAN release: 2019-02-22 Progress bar now shown workers start (#107) Socket connections now authenticated using session password (#125) Marked internal functions @keywords internal Added vignettes User Guide Technical Documentation","code":""},{"path":"/news/index.html","id":"clustermq-085","dir":"Changelog","previous_headings":"","what":"clustermq 0.8.5","title":"clustermq 0.8.5","text":"CRAN release: 2018-09-29 Added experimental support parallel foreach backend (#83) Moved templates package inst/ directory (#85) Added send_call worker evaluate arbitrary expressions (drake#501; #86) Option clustermq.scheduler now respected set package load (#88) System interrupts now handled correctly (rzmq#44; #73, #93, #97) Number workers running/total now shown progress bar (#98) Unqualified (short) host names now resolved default (#104)","code":""},{"path":"/news/index.html","id":"clustermq-084","dir":"Changelog","previous_headings":"","what":"clustermq 0.8.4","title":"clustermq 0.8.4","text":"CRAN release: 2018-04-22 Fix error qsys$reusable using n_jobs=0/local processing (#75) Scheduler-specific templates deprecated. Use clustermq.template instead Allow option clustermq.defaults fill default template values (#71) Errors worker processing now shut cleanly (#67) Progress bar now shows estimated time remaining (#66) Progress bar now also shown processing locally Memory summary now adds estimated memory R session (#69)","code":""},{"path":"/news/index.html","id":"clustermq-083","dir":"Changelog","previous_headings":"","what":"clustermq 0.8.3","title":"clustermq 0.8.3","text":"CRAN release: 2018-01-21 Support rettype function calls return type known (#59) Reduce memory requirements processing results receive Fix bug cleanup, log_worker flag working SGE/SLURM","code":""},{"path":"/news/index.html","id":"clustermq-082","dir":"Changelog","previous_headings":"","what":"clustermq 0.8.2","title":"clustermq 0.8.2","text":"CRAN release: 2017-11-30 Fix bug never-started jobs cleaned Fix bug tests leave processes port binding fails (#60) Multicore longer prints worker debug messages (#61)","code":""},{"path":"/news/index.html","id":"clustermq-081","dir":"Changelog","previous_headings":"","what":"clustermq 0.8.1","title":"clustermq 0.8.1","text":"CRAN release: 2017-11-27 Fix performance issues high number function calls (#56) Fix bug multicore workers shut properly (#58) Fix default templates SGE, LSF SLURM (misplaced quote)","code":""},{"path":"/news/index.html","id":"clustermq-080","dir":"Changelog","previous_headings":"","what":"clustermq 0.8.0","title":"clustermq 0.8.0","text":"CRAN release: 2017-11-11 Templates changed: clustermq:::worker now takes master argument Fix bug copies common_data collected gc slowly (#19) Creating workers now separated Q, enabling worker reuse (#45) Objects function environment must now exported explicitly (#47) Messages master now processed threads (#42) Added multicore qsys using parallel package (#49) New function Q_rows using data.frame rows iterated arguments (#43) Jobs now submitted array possible Job summary now report max memory reported gc (#18)","code":""},{"path":"/news/index.html","id":"clustermq-070","dir":"Changelog","previous_headings":"","what":"clustermq 0.7.0","title":"clustermq 0.7.0","text":"CRAN release: 2017-08-28 Initial release CRAN","code":""}] +[{"path":"/articles/quickstart.html","id":"installation","dir":"Articles","previous_headings":"","what":"Installation","title":"Quick Start","text":"First, need ZeroMQ system library. probably already installed system. , package manager provide : install clustermq package R CRAN: Alternatively can use remotes package install directly Github: good go! default, clustermq look sbatch (SLURM), bsub (LSF), qsub (SGE) $PATH use scheduler available. examples don’t run box, might need set scheduler explicitly.","code":"# You can skip this step on Windows and macOS, the package binary has it # On a computing cluster, we recommend to use Conda or Linuxbrew brew install zeromq # Linuxbrew, Homebrew on macOS conda install zeromq # Conda, Miniconda sudo apt-get install libzmq3-dev # Ubuntu sudo yum install zeromq-devel # Fedora pacman -S zeromq # Arch Linux install.packages('clustermq') # install.packages('remotes') remotes::install_github('mschubert/clustermq') # remotes::install_github('mschubert/clustermq', ref=\"develop\") # dev version"},{"path":"/articles/quickstart.html","id":"setting-up-the-scheduler-explicitly","dir":"Articles","previous_headings":"","what":"Setting up the scheduler explicitly","title":"Quick Start","text":"HPC cluster’s scheduler ensures computing jobs distributed available worker nodes. Hence, clustermq interfaces order computations. currently support following schedulers (either locally via SSH): Multiprocess - test calls parallelize cores using options(clustermq.scheduler=\"multiprocess\") LSF - work without setup SGE - work without setup SLURM - work without setup PBS/Torque - needs options(clustermq.scheduler=\"PBS\"/\"Torque\") via SSH - needs options(clustermq.scheduler=\"ssh\", clustermq.ssh.host=) Default submission templates provided can customized, e.g. activate compute environments containers.","code":""},{"path":"/articles/quickstart.html","id":"examples","dir":"Articles","previous_headings":"","what":"Examples","title":"Quick Start","text":"package designed distribute arbitrary function calls HPC worker nodes. , however, couple caveats observe R session running worker share local memory. simplest example function call completely self-sufficient, one argument (x) iterate : Non-iterated arguments supported const argument: function relies objects environment passed arguments, can exported using export argument: want use package function need load worker using library() call referencing package_name::: clustermq can also used parallel backend foreach. also used BiocParallel, can run packages cluster well: examples available user guide.","code":"fx = function(x) x * 2 Q(fx, x=1:3, n_jobs=1) #> Running sequentially ('LOCAL') ... #> [[1]] #> [1] 2 #> #> [[2]] #> [1] 4 #> #> [[3]] #> [1] 6 fx = function(x, y) x * 2 + y Q(fx, x=1:3, const=list(y=10), n_jobs=1) #> Running sequentially ('LOCAL') ... #> [[1]] #> [1] 12 #> #> [[2]] #> [1] 14 #> #> [[3]] #> [1] 16 fx = function(x) x * 2 + y Q(fx, x=1:3, export=list(y=10), n_jobs=1) #> Running sequentially ('LOCAL') ... #> [[1]] #> [1] 12 #> #> [[2]] #> [1] 14 #> #> [[3]] #> [1] 16 fx = function(x) { `%>%` = dplyr::`%>%` x %>% dplyr::mutate(area = Sepal.Length * Sepal.Width) %>% head() } Q(fx, x=list(iris), n_jobs=1) #> Running sequentially ('LOCAL') ... #> [[1]] #> Sepal.Length Sepal.Width Petal.Length Petal.Width Species area #> 1 5.1 3.5 1.4 0.2 setosa 17.85 #> 2 4.9 3.0 1.4 0.2 setosa 14.70 #> 3 4.7 3.2 1.3 0.2 setosa 15.04 #> 4 4.6 3.1 1.5 0.2 setosa 14.26 #> 5 5.0 3.6 1.4 0.2 setosa 18.00 #> 6 5.4 3.9 1.7 0.4 setosa 21.06 library(foreach) register_dopar_cmq(n_jobs=2, memory=1024) # accepts same arguments as `workers` foreach(i=1:3) %dopar% sqrt(i) # this will be executed as jobs #> Running sequentially ('LOCAL') ... #> [[1]] #> [1] 1 #> #> [[2]] #> [1] 1.414214 #> #> [[3]] #> [1] 1.732051 library(BiocParallel) register(DoparParam()) # after register_dopar_cmq(...) bplapply(1:3, sqrt)"},{"path":"/articles/quickstart.html","id":"usage","dir":"Articles","previous_headings":"","what":"Usage","title":"Quick Start","text":"following arguments supported Q: fun - function call. needs self-sufficient (access master environment) ... - iterated arguments passed function. one, need named const - named list non-iterated arguments passed fun export - named list objects export worker environment Behavior can fine-tuned using options : fail_on_error - Whether stop one calls returns error seed - common seed combined job number reproducible results memory - Amount memory request job (bsub -M) n_jobs - Number jobs submit function calls job_size - Number function calls per job. used combination n_jobs latter overall limit chunk_size - many calls worker process reporting back master. Default: every worker report back 100 times total full documentation available typing ?Q.","code":""},{"path":"/articles/quickstart.html","id":"comparison-to-other-packages","dir":"Articles","previous_headings":"","what":"Comparison to other packages","title":"Quick Start","text":"packages provide high-level parallelization R function calls computing cluster. thorough comparison features performance available wiki. Briefly, compare long takes different HPC scheduler tools submit, run collect function calls negligible processing time (multiplying numeric value 2). serves quantify maximum throughput can reach BatchJobs, batchtools clustermq. find BatchJobs unable process 106 calls produces reproducible RSQLite error. batchtools able process function calls, file system practically limits 106 calls. clustermq problems processing 109 calls, still faster batchtools 106 calls. short, use ClusterMQ want: one-line solution run cluster jobs minimal setup access cluster functions local Rstudio via SSH fast processing many function calls without network storage /O Use batchtools : want use mature well-tested package don’t mind arguments every call written /read disc don’t mind ’s load-balancing run-time Use Snakemake (flowr, remake, drake) : want design run pipeline different tools Don’t use batch (last updated 2013) BatchJobs (issues SQLite network-mounted storage).","code":""},{"path":[]},{"path":"/articles/technicaldocs.html","id":"base-api-and-schedulers","dir":"Articles","previous_headings":"Worker API","what":"Base API and schedulers","title":"Technical Documentation","text":"main worker functions wrapped R6 class name QSys. provides standardized API lower-level messages sent via ZeroMQ. base class derived scheduler classes add required functions submitting cleaning jobs: user-visible object worker Pool wraps , eventually allow manage different workers.","code":"+ QSys |- Multicore |- LSF + SGE |- PBS |- Torque |- etc."},{"path":[]},{"path":"/articles/technicaldocs.html","id":"creating-a-worker-pool","dir":"Articles","previous_headings":"Worker API > Workers","what":"Creating a worker pool","title":"Technical Documentation","text":"pool workers can created using workers() function, instantiates Pool object corresponding QSys-derived scheduler class. See ?workers details.","code":"# start up a pool of three workers using the default scheduler w = workers(n_jobs=3) # if we make an unclean exit for whatever reason, clean up the jobs on.exit(w$cleanup())"},{"path":"/articles/technicaldocs.html","id":"worker-startup","dir":"Articles","previous_headings":"Worker API > Workers","what":"Worker startup","title":"Technical Documentation","text":"workers started via scheduler, know machine run . start every worker TCP/IP address master socket distribute work. achieved call R common schedulers:","code":"R --no-save --no-restore -e 'clustermq:::worker(\"{{ master }}\")'"},{"path":"/articles/technicaldocs.html","id":"worker-communication","dir":"Articles","previous_headings":"Worker API > Workers","what":"Worker communication","title":"Technical Documentation","text":"master’s side, wait worker connects: can send expression evaluated worker using send method: expression (...), variables passed along call can added. batch processing clustermq usually , command work_chunk, chunk data added:","code":"msg = w$recv() # this will block until a worker is ready w$send(expression, ...) w$send(clustermq:::work_chunk(chunk, fun, const, rettype, common_seed), chunk = chunk(iter, submit_index))"},{"path":"/articles/technicaldocs.html","id":"worker-environment","dir":"Articles","previous_headings":"Worker API > Workers","what":"Worker environment","title":"Technical Documentation","text":"can add number objects worker environment using env method: also invisibly return data.frame objects currently environment. user wants inspect environment without changing can call w$env() without arguments. environment propagated workers automatically greedy fashion.","code":"w$env(object=value, ...)"},{"path":"/articles/technicaldocs.html","id":"main-event-loop","dir":"Articles","previous_headings":"Worker API","what":"Main event loop","title":"Technical Documentation","text":"Putting together event loop, get essentially implemented master. Note expression evaluate needs track origin (may change future). loop similar structure can used extend clustermq. example, done targets package.","code":"w = workers(3) on.exit(w$cleanup()) w$env(...) while (we have new work to send || jobs pending) { res = w$recv() # handle result if (more work) w$send(expression, ...) else w$send_shutdown() }"},{"path":"/articles/technicaldocs.html","id":"zeromq-message-specification","dir":"Articles","previous_headings":"","what":"ZeroMQ message specification","title":"Technical Documentation","text":"Communication master (main event loop) workers (QSys base class) organised messages. chunks serialized data sent via ZeroMQ’s protocol (ZMTP). parts message called frames.","code":""},{"path":"/articles/technicaldocs.html","id":"master---worker-communication","dir":"Articles","previous_headings":"ZeroMQ message specification","what":"Master - Worker communication","title":"Technical Documentation","text":"master requests evaluation message X frames (direct) Y proxied. handled clustermq internally. worker identity frame routing identifier delimiter frame Worker status (wlife_t) call evaluated variable name environment object yet present worker variable value using proxy, followed SEXP contains variable names proxy add forwarding worker.","code":""},{"path":"/articles/technicaldocs.html","id":"worker-evaluation","dir":"Articles","previous_headings":"ZeroMQ message specification","what":"Worker evaluation","title":"Technical Documentation","text":"worker evaluates call using R C API: error occurs evaluation returned structure class worker_error. developer wants catch errors warnings fine-grained manner, recommended add callingHandlers cmd (clustermq work work_chunk).","code":"R_tryEvalSilent(cmd, env, &err);"},{"path":"/articles/technicaldocs.html","id":"worker---master-communication","dir":"Articles","previous_headings":"ZeroMQ message specification","what":"Worker - Master communication","title":"Technical Documentation","text":"result evaluation returned message four (direct) five (proxied) frames: Worker identity frame (handled internally ZeroMQ’s ZMQ_REQ socket) Empty frame (handled internally ZeroMQ’s ZMQ_REQ socket) Worker status (wlife_t) handled internally clustermq result call (SEXP), visible user using worker via SSH, frames preceded routing identify frame handled internally ZeroMQ added peeled proxy.","code":""},{"path":"/articles/userguide.html","id":"installation","dir":"Articles","previous_headings":"","what":"Installation","title":"User Guide","text":"Install clustermq package R CRAN (including bundled ZeroMQ system library): Alternatively can use remotes package install directly Github: develop branch, introduce code changes new features. may contain bugs, poor documentation, inconveniences. branch may install times. However, feedback welcome.","code":"install.packages('clustermq') # install.packages('remotes') remotes::install_github('mschubert/clustermq') # install.packages('remotes') remotes::install_github('mschubert/clustermq', ref=\"develop\")"},{"path":"/articles/userguide.html","id":"configuration","dir":"Articles","previous_headings":"","what":"Configuration","title":"User Guide","text":"Choose preferred parallelism using: three kinds schedulers: LOCAL - sequential processing calls (default HPC scheduler found) Multiple cores machine HPC schedulers Parallel HPC schedulers can also used via SSH.","code":"options(clustermq.scheduler = \"your scheduler here\")"},{"path":"/articles/userguide.html","id":"local-parallelization","dir":"Articles","previous_headings":"Configuration","what":"Local parallelization","title":"User Guide","text":"main focus package, can use parallelize function calls locally multiple cores processes. can also useful test code submitting scheduler. Multiprocess (recommended) - Use callr package run manage multiple parallel R processes options(clustermq.scheduler=\"multiprocess\") Multicore - Uses parallel package fork current R process multiple threads options(clustermq.scheduler=\"multicore\"). sometimes causes problems (macOS, RStudio) available Windows.","code":""},{"path":"/articles/userguide.html","id":"setting-up-the-scheduler","dir":"Articles","previous_headings":"Configuration","what":"Setting up the scheduler","title":"User Guide","text":"HPC cluster’s scheduler ensures computing jobs distributed available worker nodes. Hence, clustermq interfaces order computations. default, take whichever scheduler find fall back local processing. work , cases. set scheduler explicitly, see following links: LSF - work without setup SGE - work without setup SLURM - work without setup PBS/Torque - needs options(clustermq.scheduler=\"PBS\"/\"Torque\") want another scheduler, open issue Default submission templates provided can customized, e.g. activate compute environments containers.","code":""},{"path":"/articles/userguide.html","id":"ssh-connector","dir":"Articles","previous_headings":"Configuration","what":"SSH connector","title":"User Guide","text":"reasons might prefer work computing cluster directly rather local machine instead. RStudio excellent local IDE, ’s responsive feature-rich browser-based solutions (RStudio server, Project Jupyter), avoids X forwarding issues want look plots just made. Using setup, however, lost access computing cluster. Instead, copy data , submit individual scripts jobs, aggregating data end . clustermq trying solve providing transparent SSH interface. order use clustermq local machine, package needs installed computing cluster. computing cluster, set scheduler make sure clustermq runs without problems. Note remote scheduler can LOCAL (default HPC scheduler found) SSH work. local machine, add following options ~/.Rprofile: recommend set SSH keys password-less login.","code":"# If this is set to 'LOCAL' or 'SSH' you will get the following error: # Expected PROXY_READY, received ‘PROXY_ERROR: Remote SSH QSys is not allowed’ options( clustermq.scheduler = \"multiprocess\" # or multicore, LSF, SGE, Slurm etc. ) options( clustermq.scheduler = \"ssh\", clustermq.ssh.host = \"user@host\", # use your user and host, obviously clustermq.ssh.log = \"~/cmq_ssh.log\" # log for easier debugging )"},{"path":[]},{"path":"/articles/userguide.html","id":"the-q-function","dir":"Articles","previous_headings":"Usage","what":"The Q function","title":"User Guide","text":"following arguments supported Q: fun - function call. needs self-sufficient (access master environment) ... - iterated arguments passed function. one, need named const - named list non-iterated arguments passed fun export - named list objects export worker environment Behavior can fine-tuned using options : fail_on_error - Whether stop one calls returns error seed - common seed combined job number reproducible results memory - Amount memory request job (bsub -M) n_jobs - Number jobs submit function calls job_size - Number function calls per job. used combination n_jobs latter overall limit chunk_size - many calls worker process reporting back master. Default: every worker report back 100 times total full documentation available typing ?Q.","code":""},{"path":"/articles/userguide.html","id":"examples","dir":"Articles","previous_headings":"Usage","what":"Examples","title":"User Guide","text":"package designed distribute arbitrary function calls HPC worker nodes. , however, couple caveats observe R session running worker share local memory. simplest example function call completely self-sufficient, one argument (x) iterate : Non-iterated arguments supported const argument: function relies objects environment passed arguments (including functions), can exported using export argument: want use package function need load worker using pkg argument referencing package_name:::","code":"fx = function(x) x * 2 Q(fx, x=1:3, n_jobs=1) #> Running sequentially ('LOCAL') ... #> [[1]] #> [1] 2 #> #> [[2]] #> [1] 4 #> #> [[3]] #> [1] 6 fx = function(x, y) x * 2 + y Q(fx, x=1:3, const=list(y=10), n_jobs=1) #> Running sequentially ('LOCAL') ... #> [[1]] #> [1] 12 #> #> [[2]] #> [1] 14 #> #> [[3]] #> [1] 16 fx = function(x) x * 2 + y Q(fx, x=1:3, export=list(y=10), n_jobs=1) #> Running sequentially ('LOCAL') ... #> [[1]] #> [1] 12 #> #> [[2]] #> [1] 14 #> #> [[3]] #> [1] 16 fx = function(x) { x %>% mutate(area = Sepal.Length * Sepal.Width) %>% head() } Q(fx, x=list(iris), pkgs=\"dplyr\", n_jobs=1) #> Running sequentially ('LOCAL') ... #> #> Attaching package: 'dplyr' #> The following objects are masked from 'package:stats': #> #> filter, lag #> The following objects are masked from 'package:base': #> #> intersect, setdiff, setequal, union #> [[1]] #> Sepal.Length Sepal.Width Petal.Length Petal.Width Species area #> 1 5.1 3.5 1.4 0.2 setosa 17.85 #> 2 4.9 3.0 1.4 0.2 setosa 14.70 #> 3 4.7 3.2 1.3 0.2 setosa 15.04 #> 4 4.6 3.1 1.5 0.2 setosa 14.26 #> 5 5.0 3.6 1.4 0.2 setosa 18.00 #> 6 5.4 3.9 1.7 0.4 setosa 21.06"},{"path":"/articles/userguide.html","id":"as-parallel-foreach-backend","dir":"Articles","previous_headings":"Usage","what":"As parallel foreach backend","title":"User Guide","text":"foreach package provides interface perform repeated tasks different backends. can perform function simple loops using %%: can also perform operations parallel using %dopar%: latter allows registering different handlers parallel execution, can use clustermq: BiocParallel supports foreach , means can run packages use BiocParallel cluster well via DoparParam.","code":"library(foreach) x = foreach(i=1:3) %do% sqrt(i) x = foreach(i=1:3) %dopar% sqrt(i) #> Warning: executing %dopar% sequentially: no parallel backend registered # set up the scheduler first, otherwise this will run sequentially clustermq::register_dopar_cmq(n_jobs=2, memory=1024) # this accepts same arguments as `Q` x = foreach(i=1:3) %dopar% sqrt(i) # this will be executed as jobs #> Running sequentially ('LOCAL') ... library(BiocParallel) register(DoparParam()) # after register_dopar_cmq(...) bplapply(1:3, sqrt)"},{"path":"/articles/userguide.html","id":"with-drake","dir":"Articles","previous_headings":"Usage","what":"With drake","title":"User Guide","text":"drake package enables users define dependency structure different function calls, evaluate underlying data changed. drake — , Data Frames R Make — general-purpose workflow manager data-driven tasks. rebuilds intermediate data objects dependencies change, skips work results already date. every runthrough starts scratch, completed workflows tangible evidence reproducibility. drake also supports scalability, parallel computing, smooth user experience comes setting , deploying, maintaining data science projects. can use clustermq perform calculations jobs:","code":"library(drake) load_mtcars_example() # clean(destroy = TRUE) # options(clustermq.scheduler = \"multicore\") make(my_plan, parallelism = \"clustermq\", jobs = 2, verbose = 4)"},{"path":"/articles/userguide.html","id":"options","dir":"Articles","previous_headings":"","what":"Options","title":"User Guide","text":"various configurable options mentioned throughout documentation, applicable, however, list options reference. Options can set including call options( = ) .Rprofile, calling options( = ) script interactively session. clustermq.scheduler - One supported clustermq schedulers; options \"LOCAL\", \"multiprocess\", \"multicore\", \"lsf\", \"sge\", \"slurm\", \"pbs\", \"Torque\", \"ssh\" (default HPC scheduler found $PATH, otherwise \"LOCAL\") clustermq.host - name node device constructing ZeroMQ host address (default Sys.info()[\"nodename\"]) clustermq.ssh.host - user name host connecting HPC via SSH (e.g. user@host); recommend setting SSH keys password-less login clustermq.ssh.log - Path file (SSH host) created populated logging information regarding SSH connection (e.g. \"~/cmq_ssh.log\"); helpful debugging purposes clustermq.ssh.timeout - amount time wait (seconds) SSH start-connection timing (default 5 seconds) clustermq.worker.timeout - amount time wait (seconds) master-worker communication timing (default 600 seconds) clustermq.error.timeout - amount time wait (seconds), case worker error, remaining workers finish computations shut cleanly (default min(timeout, 30) seconds) clustermq.template - Path template file submitting HPC jobs; necessary using template, otherwise default template used (default depends clustermq.scheduler) clustermq.data.warning - threshold size common data (Mb) clustermq throws warning (default 1000) clustermq.defaults - named-list default values HPC template; takes precedence defaults specified template file (default empty list (.e. list()))","code":""},{"path":[]},{"path":"/articles/userguide.html","id":"debugging-workers","dir":"Articles","previous_headings":"Troubleshooting","what":"Debugging workers","title":"User Guide","text":"Function calls evaluated workers wrapped event handlers, means even call evaluation throws error, reported back main R session. However, reasons workers might crash, case can report back. include: segfault low-level process Process kill due resource constraints (e.g. walltime) Reaching wait timeout without signal master process Probably others case, useful worker(s) create log file also include events reported back. can requested using: create file called -.log current working directory, irrespective scheduler use. can customize file name using Note case log_file template field scheduler script, hence needs present order work. default templates field included. order log worker separately, schedulers support wildcards log file names. instance: Multicore/Multiprocess: log_file=\"/path/.file.%\" SGE: log_file=\"/path/.file.\\$TASK_ID\" LSF: log_file=\"/path/.file.%\" Slurm: log_file=\"/path/.file.%\" PBS: log_file=\"/path/.file.$PBS_ARRAY_INDEX\" Torque: log_file=\"/path/.file.$PBS_ARRAYID\" scheduler documentation details available options. reporting bug includes worker crashes, please always include log file.","code":"Q(..., log_worker=TRUE) Q(..., template=list(log_file = ))"},{"path":"/articles/userguide.html","id":"ssh","dir":"Articles","previous_headings":"Troubleshooting","what":"SSH","title":"User Guide","text":"trying remote schedulers via SSH, make sure scheduler works first connect cluster run job . terminal stuck make sure step SSH connection works typing following commands local terminal make sure don’t get errors warnings step: get Command found: R error, make sure $PATH set correctly ~/.bash_profile /~/.bashrc (depending cluster config might need either). may also need modify SSH template load R module conda environment. get SSH warning error try ssh -v enable verbose output. forward works, run following local R session (ideally also command-line R, RStudio): create log file remote server contain errors might occurred ssh_proxy startup. ssh_proxy startup fails local machine error server log show errors, can try increasing timeout: can happen SSH startup template includes additional steps starting R, activating module conda environment, confirm connection via two-factor authentication.","code":"Connecting via SSH ... # test your ssh login that you set up in ~/.ssh/config # if this fails you have not set up SSH correctly ssh # test port forwarding from 54709 remote to 6687 local (ports are random) # if the fails you will not be able to use clustermq via SSH ssh -R 54709:localhost:6687 R --vanilla options(clustermq.scheduler = \"ssh\", clustermq.ssh.log = \"~/ssh_proxy.log\") Q(identity, x=1, n_jobs=1) Remote R process did not respond after 5 seconds. Check your SSH server log. options(clustermq.ssh.timeout = 30) # in seconds"},{"path":[]},{"path":"/articles/userguide.html","id":"environments-for-workers","dir":"Articles","previous_headings":"Environments","what":"Environments for workers","title":"User Guide","text":"cases, may necessary activate specific computing environment scheduler jobs prior starting worker. can , instance, R installed specific environment container. Examples environments containers : Bash module environments Conda environments Docker/Singularity containers possible activate job submission script (.e., template file). widely untested, look following LSF scheduler (analogous others): template still needs filled, example need pass either set via .Rprofile option:","code":"#BSUB-J {{ job_name }}[1-{{ n_jobs }}] # name of the job / array jobs #BSUB-o {{ log_file | /dev/null }} # stdout + stderr #BSUB-M {{ memory | 4096 }} # Memory requirements in Mbytes #BSUB-R rusage[mem={{ memory | 4096 }}] # Memory requirements in Mbytes ##BSUB-q default # name of the queue (uncomment) ##BSUB-W {{ walltime | 6:00 }} # walltime (uncomment) module load {{ bashenv | default_bash_env }} # or: source activate {{ conda | default_conda_env_name }} # or: your environment activation command ulimit -v $(( 1024 * {{ memory | 4096 }} )) CMQ_AUTH={{ auth }} R --no-save --no-restore -e 'clustermq:::worker(\"{{ master }}\")' Q(..., template=list(bashenv=\"my environment name\")) options( clustermq.defaults = list(bashenv=\"my default env\") )"},{"path":"/articles/userguide.html","id":"running-master-inside-containers","dir":"Articles","previous_headings":"Environments","what":"Running master inside containers","title":"User Guide","text":"master process inside container, accessing HPC scheduler difficult. Containers, including singularity docker, isolate processes inside container host. R process able submit job scheduler found. Note HPC node running master process must allowed submit jobs. HPC systems allow compute nodes submit jobs. case, may need run master process login node, discuss issue system administrator. container binary compatible host, may able bind scheduler executable container. example, PBS might look something like: working example binding SLURM CentOS 7 container image CentOS 7 host available https://groups.google.com//lbl.gov/d/msg/singularity/syLcsIWWzdo/NZvF2Ud2AAAJ Alternatively, can create script uses SSH execute scheduler login node. , need SSH client container, keys set password-less login, create script call scheduler login node via ssh (e.g. ~/bin/qsub SGE/PBS/Torque, bsub LSF sbatch Slurm): Make sure script executable, bind/copy container somewhere $PATH. Home directories bound default singularity.","code":"#PBS directives ... module load singularity SINGULARITYENV_APPEND_PATH=/opt/pbs/bin singularity exec --bind /opt/pbs/bin r_image.sif Rscript master_script.R #!/bin/bash ssh -i ~/.ssh/ ${PBS_O_HOST:-\"no_host_not_in_a_pbs_job\"} qsub \"$@\" chmod u+x ~/bin/qsub SINGULARITYENV_APPEND_PATH=~/bin"},{"path":[]},{"path":"/articles/userguide.html","id":"LSF","dir":"Articles","previous_headings":"Scheduler templates","what":"LSF","title":"User Guide","text":"~/.Rprofile computing cluster, set following options: option clustermq.template point LSF template file like one (needed want supply template rather using default). file, #BSUB-* defines command-line arguments bsub program. Memory: defined BSUB-M BSUB-R. Check local setup memory values supplied MiB KiB, default 4096 requesting memory calling Q() Queue: BSUB-q default. Use queue name default. likely exist system, choose right name (comment line additional #) Walltime: BSUB-W {{ walltime }}. Set maximum time job allowed run killed. default disable line. enable , enter fixed value pass walltime argument function call. way written, use 6 hours arguemnt given. options, see LSF documentation add via #BSUB-* (* represents argument) change identifiers curly braces ({{ ... }}), used fill right variables done, package use settings longer warn missing options.","code":"options( clustermq.scheduler = \"lsf\", clustermq.template = \"/path/to/file/below\" # if using your own template ) #BSUB-J {{ job_name }}[1-{{ n_jobs }}] # name of the job / array jobs #BSUB-n {{ cores | 1 }} # number of cores to use per job #BSUB-o {{ log_file | /dev/null }} # stdout + stderr; %I for array index #BSUB-M {{ memory | 4096 }} # Memory requirements in Mbytes #BSUB-R rusage[mem={{ memory | 4096 }}] # Memory requirements in Mbytes ##BSUB-q default # name of the queue (uncomment) ##BSUB-W {{ walltime | 6:00 }} # walltime (uncomment) ulimit -v $(( 1024 * {{ memory | 4096 }} )) CMQ_AUTH={{ auth }} R --no-save --no-restore -e 'clustermq:::worker(\"{{ master }}\")'"},{"path":"/articles/userguide.html","id":"SGE","dir":"Articles","previous_headings":"Scheduler templates","what":"SGE","title":"User Guide","text":"~/.Rprofile computing cluster, set following options: option clustermq.template point SGE template file like one (needed want supply template rather using default). file, #$-* defines command-line arguments qsub program. Queue: $ -q default. Use queue name default. likely exist system, choose right name (comment line additional #) options, see SGE documentation. change identifiers curly braces ({{ ... }}), used fill right variables. done, package use settings longer warn missing options.","code":"options( clustermq.scheduler = \"sge\", clustermq.template = \"/path/to/file/below\" # if using your own template ) #$ -N {{ job_name }} # job name #$ -q default # submit to queue named \"default\" #$ -j y # combine stdout/error in one file #$ -o {{ log_file | /dev/null }} # output file #$ -cwd # use pwd as work dir #$ -V # use environment variable #$ -t 1-{{ n_jobs }} # submit jobs as array #$ -pe {{ cores | 1 }} # number of cores to use per job ulimit -v $(( 1024 * {{ memory | 4096 }} )) CMQ_AUTH={{ auth }} R --no-save --no-restore -e 'clustermq:::worker(\"{{ master }}\")'"},{"path":"/articles/userguide.html","id":"SLURM","dir":"Articles","previous_headings":"Scheduler templates","what":"SLURM","title":"User Guide","text":"~/.Rprofile computing cluster, set following options: option clustermq.template point SLURM template file like one (needed want supply template rather using default). file, #SBATCH defines command-line arguments sbatch program. Queue: SBATCH --partition default. Use queue name default. likely exist system, choose right name (comment line additional #) options, see SLURM documentation. change identifiers curly braces ({{ ... }}), used fill right variables. done, package use settings longer warn missing options.","code":"options( clustermq.scheduler = \"slurm\", clustermq.template = \"/path/to/file/below\" # if using your own template ) #!/bin/sh #SBATCH --job-name={{ job_name }} #SBATCH --partition=default #SBATCH --output={{ log_file | /dev/null }} # you can add .%a for array index #SBATCH --error={{ log_file | /dev/null }} #SBATCH --mem-per-cpu={{ memory | 4096 }} #SBATCH --array=1-{{ n_jobs }} #SBATCH --cpus-per-task={{ cores | 1 }} ulimit -v $(( 1024 * {{ memory | 4096 }} )) CMQ_AUTH={{ auth }} R --no-save --no-restore -e 'clustermq:::worker(\"{{ master }}\")'"},{"path":"/articles/userguide.html","id":"PBS","dir":"Articles","previous_headings":"Scheduler templates","what":"PBS","title":"User Guide","text":"~/.Rprofile computing cluster, set following options: option clustermq.template point PBS template file like one (needed want supply template rather using default). file, #PBS-* defines command-line arguments qsub program. Queue: #PBS-q default. Use queue name default. likely exist system, choose right name (comment line additional #) options, see PBS documentation. change identifiers curly braces ({{ ... }}), used fill right variables. done, package use settings longer warn missing options.","code":"options( clustermq.scheduler = \"pbs\", clustermq.template = \"/path/to/file/below\" # if using your own template ) #PBS -N {{ job_name }} #PBS -J 1-{{ n_jobs }} #PBS -l select=1:ncpus={{ cores | 1 }}:mpiprocs={{ cores | 1 }}:mem={{ memory | 4096 }}MB #PBS -l walltime={{ walltime | 12:00:00 }} #PBS -o {{ log_file | /dev/null }} #PBS -j oe #PBS -q default ulimit -v $(( 1024 * {{ memory | 4096 }} )) CMQ_AUTH={{ auth }} R --no-save --no-restore -e 'clustermq:::worker(\"{{ master }}\")'"},{"path":"/articles/userguide.html","id":"Torque","dir":"Articles","previous_headings":"Scheduler templates","what":"Torque","title":"User Guide","text":"~/.Rprofile computing cluster, set following options: option clustermq.template point Torque template file like one (needed want supply template rather using default). file, #PBS-* defines command-line arguments qsub program. Queue: #PBS -q default. Use queue name default. likely exist system, choose right name (comment line additional #) options, see Torque documentation. change identifiers curly braces ({{ ... }}), used fill right variables. done, package use settings longer warn missing options.","code":"options(clustermq.scheduler = \"Torque\", clustermq.template = \"/path/to/file/below\" # if using your own template ) #PBS -N {{ job_name }} #PBS -l nodes={{ n_jobs }}:ppn={{ cores | 1 }},walltime={{ walltime | 12:00:00 }} #PBS -o {{ log_file | /dev/null }} #PBS -q default #PBS -j oe ulimit -v $(( 1024 * {{ memory | 4096 }} )) CMQ_AUTH={{ auth }} R --no-save --no-restore -e 'clustermq:::worker(\"{{ master }}\")'"},{"path":"/articles/userguide.html","id":"ssh-template","dir":"Articles","previous_headings":"Scheduler templates","what":"SSH","title":"User Guide","text":"SSH scheduler, can access remote schedulers via SSH. want use , first make sure real scheduler running manually connecting HPC environment. default template shown . R HPC $PATH, may need specify path load required bash modules/conda environments.","code":"options(clustermq.scheduler = \"ssh\", clustermq.ssh.host = \"myhost\", # set this up in your local ~/.ssh/config clustermq.ssh.log = \"~/ssh_proxy.log\", # log file on your HPC clustermq.ssh.timeout = 30, # if changing the default connection timeout clustermq.template = \"/path/to/file/below\" # if using your own template ) ssh -o \"ExitOnForwardFailure yes\" -f \\ -R {{ ctl_port }}:localhost:{{ local_port }} \\ -R {{ job_port }}:localhost:{{ fwd_port }} \\ {{ ssh_host }} \\ \"R --no-save --no-restore -e \\ 'clustermq:::ssh_proxy(ctl={{ ctl_port }}, job={{ job_port }})' \\ > {{ ssh_log | /dev/null }} 2>&1\""},{"path":"/authors.html","id":null,"dir":"","previous_headings":"","what":"Authors","title":"Authors and Citation","text":"Michael Schubert. Author, maintainer, copyright holder. ZeroMQ authors. Author, copyright holder. source files 'src/libzmq' 'src/cppzmq'","code":""},{"path":"/authors.html","id":"citation","dir":"","previous_headings":"","what":"Citation","title":"Authors and Citation","text":"Schubert, M. clustermq enables efficient parallelisation genomic analyses. Bioinformatics (2019). doi:10.1093/bioinformatics/btz284","code":"@Article{, title = {clustermq enables efficient parallelisation of genomic analyses}, author = {Michael Schubert}, journal = {Bioinformatics}, month = {May}, year = {2019}, language = {en}, doi = {10.1093/bioinformatics/btz284}, url = {https://github.com/mschubert/clustermq}, }"},{"path":"/index.html","id":"clustermq-send-r-function-calls-as-cluster-jobs","dir":"","previous_headings":"","what":"Evaluate Function Calls on HPC Schedulers (LSF, SGE, SLURM, PBS/Torque)","title":"Evaluate Function Calls on HPC Schedulers (LSF, SGE, SLURM, PBS/Torque)","text":"package allow send function calls jobs computing cluster minimal interface provided Q function: Computations done entirely network without temporary files network-mounted storage, strain file system apart starting R per job. calculations load-balanced, .e. workers get jobs done faster also receive function calls work . especially useful calls return time, one worker high load. Browse vignettes : User Guide Technical Documentation","code":"# load the library and create a simple function library(clustermq) fx = function(x) x * 2 # queue the function call on your scheduler Q(fx, x=1:3, n_jobs=1) # list(2,4,6)"},{"path":"/index.html","id":"installation","dir":"","previous_headings":"","what":"Installation","title":"Evaluate Function Calls on HPC Schedulers (LSF, SGE, SLURM, PBS/Torque)","text":"Install clustermq package R CRAN (including bundled ZeroMQ system library): Alternatively can use remotes package install directly Github:","code":"install.packages('clustermq') # install.packages('remotes') remotes::install_github('mschubert/clustermq') # remotes::install_github('mschubert/clustermq', ref=\"develop\") # dev version"},{"path":"/index.html","id":"schedulers","dir":"","previous_headings":"","what":"Schedulers","title":"Evaluate Function Calls on HPC Schedulers (LSF, SGE, SLURM, PBS/Torque)","text":"HPC cluster’s scheduler ensures computing jobs distributed available worker nodes. Hence, clustermq interfaces order computations. currently support following schedulers (either locally via SSH): Multiprocess - test calls parallelize cores using options(clustermq.scheduler=\"multiprocess\") LSF - work without setup SGE - work without setup SLURM - work without setup PBS/Torque - needs options(clustermq.scheduler=\"PBS\"/\"Torque\") via SSH - needs options(clustermq.scheduler=\"ssh\", clustermq.ssh.host=) Default submission templates provided can customized, e.g. activate compute environments containers.","code":""},{"path":"/index.html","id":"usage","dir":"","previous_headings":"","what":"Usage","title":"Evaluate Function Calls on HPC Schedulers (LSF, SGE, SLURM, PBS/Torque)","text":"common arguments Q : fun - function call. needs self-sufficient (access master environment) ... - iterated arguments passed function. one, need named const - named list non-iterated arguments passed fun export - named list objects export worker environment documentation arguments can accessed typing ?Q. Examples using const export : clustermq can also used parallel backend foreach. also used BiocParallel, can run packages cluster well: examples available user guide.","code":"# adding a constant argument fx = function(x, y) x * 2 + y Q(fx, x=1:3, const=list(y=10), n_jobs=1) # exporting an object to workers fx = function(x) x * 2 + y Q(fx, x=1:3, export=list(y=10), n_jobs=1) library(foreach) register_dopar_cmq(n_jobs=2, memory=1024) # see `?workers` for arguments foreach(i=1:3) %dopar% sqrt(i) # this will be executed as jobs library(BiocParallel) register(DoparParam()) # after register_dopar_cmq(...) bplapply(1:3, sqrt)"},{"path":"/index.html","id":"comparison-to-other-packages","dir":"","previous_headings":"","what":"Comparison to other packages","title":"Evaluate Function Calls on HPC Schedulers (LSF, SGE, SLURM, PBS/Torque)","text":"packages provide high-level parallelization R function calls computing cluster. compared clustermq BatchJobs batchtools processing many short-running jobs, found approximately 1000x less overhead cost. short, use clustermq want: one-line solution run cluster jobs minimal setup access cluster functions local Rstudio via SSH fast processing many function calls without network storage /O Use batchtools : want use mature well-tested package don’t mind arguments every call written /read disc don’t mind ’s load-balancing run-time Use Snakemake drake : want design run workflow HPC Don’t use batch (last updated 2013) BatchJobs (issues SQLite network-mounted storage).","code":""},{"path":"/index.html","id":"contributing","dir":"","previous_headings":"","what":"Contributing","title":"Evaluate Function Calls on HPC Schedulers (LSF, SGE, SLURM, PBS/Torque)","text":"use Github’s Issue Tracker coordinate development clustermq. Contributions welcome come many different forms, shapes, sizes. include, limited : Questions: welcome ask questions something clear User guide. Bug reports: Let us know something work expected. sure include self-contained Minimal Reproducible Example set log_worker=TRUE. Code contributions: look good first issue tag. Please discuss anything complicated putting lot work , ’m happy help get started.","code":""},{"path":"/index.html","id":"citation","dir":"","previous_headings":"","what":"Citation","title":"Evaluate Function Calls on HPC Schedulers (LSF, SGE, SLURM, PBS/Torque)","text":"project part academic work, evaluated citations. like able continue working research support tools like clustermq, please cite article using publications: M Schubert. clustermq enables efficient parallelisation genomic analyses. Bioinformatics (2019). doi:10.1093/bioinformatics/btz284","code":""},{"path":"/reference/LOCAL.html","id":null,"dir":"Reference","previous_headings":"","what":"Placeholder for local processing — LOCAL","title":"Placeholder for local processing — LOCAL","text":"Mainly tests pass without setting scheduler","code":""},{"path":"/reference/LSF.html","id":null,"dir":"Reference","previous_headings":"","what":"LSF scheduler functions — LSF","title":"LSF scheduler functions — LSF","text":"Derives QSys provide LSF-specific functions","code":""},{"path":"/reference/MULTICORE.html","id":null,"dir":"Reference","previous_headings":"","what":"Process on multiple cores on one machine — MULTICORE","title":"Process on multiple cores on one machine — MULTICORE","text":"Derives QSys provide multicore-specific functions","code":""},{"path":"/reference/MULTIPROCESS.html","id":null,"dir":"Reference","previous_headings":"","what":"Process on multiple processes on one machine — MULTIPROCESS","title":"Process on multiple processes on one machine — MULTIPROCESS","text":"Derives QSys provide callr-specific functions","code":""},{"path":"/reference/Pool.html","id":null,"dir":"Reference","previous_headings":"","what":"Class for basic queuing system functions — Pool","title":"Class for basic queuing system functions — Pool","text":"Provides basic functions needed communicate machines abstract functions rZMQ scheduler implementations can rely higher level functionality","code":""},{"path":"/reference/Q.html","id":null,"dir":"Reference","previous_headings":"","what":"Queue function calls on the cluster — Q","title":"Queue function calls on the cluster — Q","text":"Queue function calls cluster","code":""},{"path":"/reference/Q.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Queue function calls on the cluster — Q","text":"","code":"Q( fun, ..., const = list(), export = list(), pkgs = c(), seed = 128965, memory = NULL, template = list(), n_jobs = NULL, job_size = NULL, split_array_by = -1, rettype = \"list\", fail_on_error = TRUE, workers = NULL, log_worker = FALSE, chunk_size = NA, timeout = Inf, max_calls_worker = Inf, verbose = TRUE )"},{"path":"/reference/Q.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Queue function calls on the cluster — Q","text":"fun function call ... Objects iterated function call const list constant arguments passed function call export List objects exported worker pkgs Character vector packages load worker seed seed set function call memory Short template=list(memory=value) template named list values fill template n_jobs number LSF jobs submit; upper limit jobs job_size given well job_size number function calls per job split_array_by dimension number split arrays `...`; default: last rettype Return type function call (vector type 'list') fail_on_error error occurs workers, continue fail? workers Optional instance QSys representing worker pool log_worker Write log file worker chunk_size Number function calls chunk together defaults 100 chunks per worker max. 10 kb per chunk timeout Maximum time seconds wait worker (default: Inf) max_calls_worker Maxmimum number function calls sent one worker verbose Print status messages progress bar (default: TRUE)","code":""},{"path":"/reference/Q.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Queue function calls on the cluster — Q","text":"list whatever `fun` returned","code":""},{"path":"/reference/Q.html","id":"ref-examples","dir":"Reference","previous_headings":"","what":"Examples","title":"Queue function calls on the cluster — Q","text":"","code":"if (FALSE) { # Run a simple multiplication for numbers 1 to 3 on a worker node fx = function(x) x * 2 Q(fx, x=1:3, n_jobs=1) # list(2,4,6) # Run a mutate() call in dplyr on a worker node iris %>% mutate(area = Q(`*`, e1=Sepal.Length, e2=Sepal.Width, n_jobs=1)) # iris with an additional column 'area' }"},{"path":"/reference/QSys.html","id":null,"dir":"Reference","previous_headings":"","what":"Class for basic queuing system functions — QSys","title":"Class for basic queuing system functions — QSys","text":"Provides basic functions needed communicate machines abstract functions rZMQ scheduler implementations can rely higher level functionality","code":""},{"path":"/reference/Q_rows.html","id":null,"dir":"Reference","previous_headings":"","what":"Queue function calls defined by rows in a data.frame — Q_rows","title":"Queue function calls defined by rows in a data.frame — Q_rows","text":"Queue function calls defined rows data.frame","code":""},{"path":"/reference/Q_rows.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Queue function calls defined by rows in a data.frame — Q_rows","text":"","code":"Q_rows( df, fun, const = list(), export = list(), pkgs = c(), seed = 128965, memory = NULL, template = list(), n_jobs = NULL, job_size = NULL, rettype = \"list\", fail_on_error = TRUE, workers = NULL, log_worker = FALSE, chunk_size = NA, timeout = Inf, max_calls_worker = Inf, verbose = TRUE )"},{"path":"/reference/Q_rows.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Queue function calls defined by rows in a data.frame — Q_rows","text":"df data.frame iterated arguments fun function call const list constant arguments passed function call export List objects exported worker pkgs Character vector packages load worker seed seed set function call memory Short template=list(memory=value) template named list values fill template n_jobs number LSF jobs submit; upper limit jobs job_size given well job_size number function calls per job rettype Return type function call (vector type 'list') fail_on_error error occurs workers, continue fail? workers Optional instance QSys representing worker pool log_worker Write log file worker chunk_size Number function calls chunk together defaults 100 chunks per worker max. 10 kb per chunk timeout Maximum time seconds wait worker (default: Inf) max_calls_worker Maxmimum number function calls sent one worker verbose Print status messages progress bar (default: TRUE)","code":""},{"path":"/reference/Q_rows.html","id":"ref-examples","dir":"Reference","previous_headings":"","what":"Examples","title":"Queue function calls defined by rows in a data.frame — Q_rows","text":"","code":"if (FALSE) { # Run a simple multiplication for data frame columns x and y on a worker node fx = function (x, y) x * y df = data.frame(x = 5, y = 10) Q_rows(df, fx, job_size = 1) # [1] 50 # Q_rows also matches the names of a data frame with the function arguments fx = function (x, y) x - y df = data.frame(y = 5, x = 10) Q_rows(df, fx, job_size = 1) # [1] 5 }"},{"path":"/reference/SGE.html","id":null,"dir":"Reference","previous_headings":"","what":"SGE scheduler functions — SGE","title":"SGE scheduler functions — SGE","text":"Derives QSys provide SGE-specific functions","code":""},{"path":"/reference/SLURM.html","id":null,"dir":"Reference","previous_headings":"","what":"SLURM scheduler functions — SLURM","title":"SLURM scheduler functions — SLURM","text":"Derives QSys provide SLURM-specific functions","code":""},{"path":"/reference/SSH.html","id":null,"dir":"Reference","previous_headings":"","what":"SSH scheduler functions — SSH","title":"SSH scheduler functions — SSH","text":"Derives QSys provide SSH-specific functions","code":""},{"path":"/reference/check_args.html","id":null,"dir":"Reference","previous_headings":"","what":"Function to check arguments with which Q() is called — check_args","title":"Function to check arguments with which Q() is called — check_args","text":"Function check arguments Q() called","code":""},{"path":"/reference/check_args.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Function to check arguments with which Q() is called — check_args","text":"","code":"check_args(fun, iter, const = list())"},{"path":"/reference/check_args.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Function to check arguments with which Q() is called — check_args","text":"fun function call iter Objects iterated function call const list constant arguments passed function call","code":""},{"path":"/reference/check_args.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Function to check arguments with which Q() is called — check_args","text":"Processed iterated argument list 'iter' list","code":""},{"path":"/reference/chunk.html","id":null,"dir":"Reference","previous_headings":"","what":"Subset index chunk for processing — chunk","title":"Subset index chunk for processing — chunk","text":"'attr' `[.data.frame` takes much CPU time","code":""},{"path":"/reference/chunk.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Subset index chunk for processing — chunk","text":"","code":"chunk(x, i)"},{"path":"/reference/chunk.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Subset index chunk for processing — chunk","text":"x Index data.frame Rows subset","code":""},{"path":"/reference/chunk.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Subset index chunk for processing — chunk","text":"x[,]","code":""},{"path":"/reference/clustermq-package.html","id":null,"dir":"Reference","previous_headings":"","what":"Evaluate Function Calls on HPC Schedulers (LSF, SGE, SLURM) — clustermq-package","title":"Evaluate Function Calls on HPC Schedulers (LSF, SGE, SLURM) — clustermq-package","text":"Provides Q function send arbitrary function calls workers HPC schedulers without relying network-mounted storage. Allows using remote schedulers via SSH.","code":""},{"path":"/reference/clustermq-package.html","id":"details","dir":"Reference","previous_headings":"","what":"Details","title":"Evaluate Function Calls on HPC Schedulers (LSF, SGE, SLURM) — clustermq-package","text":"hood, submit cluster job connects master via TCP master send function argument chunks worker worker return results master everything done get back result Computations done entirely network without temporary files network-mounted storage, strain file system apart starting R per job. removes biggest bottleneck distributed computing. Using approach, can easily load-balancing, .e. workers get jobs done faster also receive function calls work . especially useful calls return time, one worker high load. detailed usage instructions, see documentation Q function.","code":""},{"path":[]},{"path":"/reference/clustermq-package.html","id":"author","dir":"Reference","previous_headings":"","what":"Author","title":"Evaluate Function Calls on HPC Schedulers (LSF, SGE, SLURM) — clustermq-package","text":"Maintainer: Michael Schubert mschu.dev@gmail.com (ORCID) [copyright holder] Authors: ZeroMQ authors (source files 'src/libzmq' 'src/cppzmq') [copyright holder]","code":""},{"path":"/reference/cmq_foreach.html","id":null,"dir":"Reference","previous_headings":"","what":"clustermq foreach handler — cmq_foreach","title":"clustermq foreach handler — cmq_foreach","text":"clustermq foreach handler","code":""},{"path":"/reference/cmq_foreach.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"clustermq foreach handler — cmq_foreach","text":"","code":"cmq_foreach(obj, expr, envir, data)"},{"path":"/reference/cmq_foreach.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"clustermq foreach handler — cmq_foreach","text":"obj Returned foreach::foreach, containing following variables: args : Arguments passed, call argnames: character vector arguments passed evalenv : Environment evaluate arguments export : character vector variable names export nodes packages: character vector required packages verbose : whether print status messages [logical] errorHandling: string function name call error , e.g. \"stop\" expr R expression curly braces envir Environment evaluate arguments data Common arguments passed register_dopcar_cmq(), e.g. n_jobs","code":""},{"path":"/reference/dot-onAttach.html","id":null,"dir":"Reference","previous_headings":"","what":"Report queueing system on package attach if not set — .onAttach","title":"Report queueing system on package attach if not set — .onAttach","text":"Report queueing system package attach set","code":""},{"path":"/reference/dot-onAttach.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Report queueing system on package attach if not set — .onAttach","text":"","code":".onAttach(libname, pkgname)"},{"path":"/reference/dot-onAttach.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Report queueing system on package attach if not set — .onAttach","text":"libname default arg compatibility pkgname default arg compatibility","code":""},{"path":"/reference/dot-onLoad.html","id":null,"dir":"Reference","previous_headings":"","what":"Select the queueing system on package loading — .onLoad","title":"Select the queueing system on package loading — .onLoad","text":"done setting variable 'qsys' package environment object contains desired queueing system.","code":""},{"path":"/reference/dot-onLoad.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Select the queueing system on package loading — .onLoad","text":"","code":".onLoad(libname, pkgname)"},{"path":"/reference/dot-onLoad.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Select the queueing system on package loading — .onLoad","text":"libname default arg compatibility pkgname default arg compatibility","code":""},{"path":"/reference/fill_template.html","id":null,"dir":"Reference","previous_headings":"","what":"Fill a template string with supplied values — fill_template","title":"Fill a template string with supplied values — fill_template","text":"Fill template string supplied values","code":""},{"path":"/reference/fill_template.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Fill a template string with supplied values — fill_template","text":"","code":"fill_template(template, values, required = c())"},{"path":"/reference/fill_template.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Fill a template string with supplied values — fill_template","text":"template character string submission template values named list key-value pairs required Keys must present template (default: none)","code":""},{"path":"/reference/fill_template.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Fill a template string with supplied values — fill_template","text":"template placeholder fields replaced values","code":""},{"path":"/reference/host.html","id":null,"dir":"Reference","previous_headings":"","what":"Construct the ZeroMQ host address — host","title":"Construct the ZeroMQ host address — host","text":"Construct ZeroMQ host address","code":""},{"path":"/reference/host.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Construct the ZeroMQ host address — host","text":"","code":"host( node = getOption(\"clustermq.host\", Sys.info()[\"nodename\"]), ports = 6000:9999, n = 100 )"},{"path":"/reference/host.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Construct the ZeroMQ host address — host","text":"node Node device name ports Range ports consider n many addresses return","code":""},{"path":"/reference/host.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Construct the ZeroMQ host address — host","text":"possible addresses character vector","code":""},{"path":"/reference/master.html","id":null,"dir":"Reference","previous_headings":"","what":"Master controlling the workers — master","title":"Master controlling the workers — master","text":"exchanging messages master workers works following way: * submitted job know start * starts, sends message list(id=0) indicating ready * send function definition common data * also send first data set work * get id > 0, result store * send next data set/index work * computatons complete, send id=0 worker * responds id=-1 (usage stats) shuts ","code":""},{"path":"/reference/master.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Master controlling the workers — master","text":"","code":"master( pool, iter, rettype = \"list\", fail_on_error = TRUE, chunk_size = NA, timeout = Inf, max_calls_worker = Inf, verbose = TRUE )"},{"path":"/reference/master.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Master controlling the workers — master","text":"pool Instance Pool object iter Objects iterated function call rettype Return type function fail_on_error error occurs workers, continue fail? chunk_size Number function calls chunk together defaults 100 chunks per worker max. 500 kb per chunk timeout Maximum time seconds wait worker (default: Inf) max_calls_worker Maxmimum number function calls sent one worker verbose Print progress messages","code":""},{"path":"/reference/master.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Master controlling the workers — master","text":"list whatever `fun` returned","code":""},{"path":"/reference/msg_fmt.html","id":null,"dir":"Reference","previous_headings":"","what":"Message format for logging — msg_fmt","title":"Message format for logging — msg_fmt","text":"Message format logging","code":""},{"path":"/reference/msg_fmt.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Message format for logging — msg_fmt","text":"","code":"msg_fmt(verbose = TRUE)"},{"path":"/reference/register_dopar_cmq.html","id":null,"dir":"Reference","previous_headings":"","what":"Register clustermq as `foreach` parallel handler — register_dopar_cmq","title":"Register clustermq as `foreach` parallel handler — register_dopar_cmq","text":"Register clustermq `foreach` parallel handler","code":""},{"path":"/reference/register_dopar_cmq.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Register clustermq as `foreach` parallel handler — register_dopar_cmq","text":"","code":"register_dopar_cmq(...)"},{"path":"/reference/register_dopar_cmq.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Register clustermq as `foreach` parallel handler — register_dopar_cmq","text":"... List arguments passed `Q` function, e.g. n_jobs","code":""},{"path":"/reference/ssh_proxy.html","id":null,"dir":"Reference","previous_headings":"","what":"SSH proxy for different schedulers — ssh_proxy","title":"SSH proxy for different schedulers — ssh_proxy","text":"call manually, SSH qsys ","code":""},{"path":"/reference/ssh_proxy.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"SSH proxy for different schedulers — ssh_proxy","text":"","code":"ssh_proxy(fwd_port, qsys_id = qsys_default)"},{"path":"/reference/ssh_proxy.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"SSH proxy for different schedulers — ssh_proxy","text":"fwd_port port master address connect (remote end reverse tunnel) qsys_id Character string QSys class use","code":""},{"path":"/reference/summarize_result.html","id":null,"dir":"Reference","previous_headings":"","what":"Print a summary of errors and warnings that occurred during processing — summarize_result","title":"Print a summary of errors and warnings that occurred during processing — summarize_result","text":"Print summary errors warnings occurred processing","code":""},{"path":"/reference/summarize_result.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Print a summary of errors and warnings that occurred during processing — summarize_result","text":"","code":"summarize_result( result, n_errors, n_warnings, cond_msgs, at = length(result), fail_on_error = TRUE )"},{"path":"/reference/summarize_result.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Print a summary of errors and warnings that occurred during processing — summarize_result","text":"result list vector processing result n_errors many errors occurred n_warnings many warnings occurred cond_msgs Error warnings messages, display first 50 many calls procesed point fail_on_error Stop error(s) occurred","code":""},{"path":"/reference/vec_lookup.html","id":null,"dir":"Reference","previous_headings":"","what":"Lookup table for return types to vector NAs — vec_lookup","title":"Lookup table for return types to vector NAs — vec_lookup","text":"Lookup table return types vector NAs","code":""},{"path":"/reference/vec_lookup.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Lookup table for return types to vector NAs — vec_lookup","text":"","code":"vec_lookup"},{"path":"/reference/vec_lookup.html","id":"format","dir":"Reference","previous_headings":"","what":"Format","title":"Lookup table for return types to vector NAs — vec_lookup","text":"object class list length 9.","code":""},{"path":"/reference/work_chunk.html","id":null,"dir":"Reference","previous_headings":"","what":"Function to process a chunk of calls — work_chunk","title":"Function to process a chunk of calls — work_chunk","text":"chunk comes encapsulated data.frame","code":""},{"path":"/reference/work_chunk.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Function to process a chunk of calls — work_chunk","text":"","code":"work_chunk( df, fun, const = list(), rettype = \"list\", common_seed = NULL, progress = FALSE )"},{"path":"/reference/work_chunk.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Function to process a chunk of calls — work_chunk","text":"df data.frame call IDs rownames arguments columns fun function call const Constant arguments passed call rettype Return type function common_seed seed offset common function calls progress Logical indicated whether display progress bar","code":""},{"path":"/reference/work_chunk.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Function to process a chunk of calls — work_chunk","text":"list call results (try-error failed)","code":""},{"path":"/reference/worker.html","id":null,"dir":"Reference","previous_headings":"","what":"R worker submitted as cluster job — worker","title":"R worker submitted as cluster job — worker","text":"call manually, master ","code":""},{"path":"/reference/worker.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"R worker submitted as cluster job — worker","text":"","code":"worker(master, ..., verbose = TRUE, context = NULL)"},{"path":"/reference/worker.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"R worker submitted as cluster job — worker","text":"master master address (tcp://ip:port) ... Catch-break older template values (ignored) verbose Whether print debug messages context ZeroMQ context (internal testing)","code":""},{"path":"/reference/workers.html","id":null,"dir":"Reference","previous_headings":"","what":"Creates a pool of workers — workers","title":"Creates a pool of workers — workers","text":"Creates pool workers","code":""},{"path":"/reference/workers.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Creates a pool of workers — workers","text":"","code":"workers( n_jobs, data = NULL, reuse = TRUE, template = list(), log_worker = FALSE, qsys_id = getOption(\"clustermq.scheduler\", qsys_default), verbose = FALSE, ... )"},{"path":"/reference/workers.html","id":"arguments","dir":"Reference","previous_headings":"","what":"Arguments","title":"Creates a pool of workers — workers","text":"n_jobs Number jobs submit (0 implies local processing) data Set common data (function, constant args, seed) reuse Whether workers reusable get shut call template named list values fill template log_worker Write log file worker qsys_id Character string QSys class use verbose Print message worker startup ... Additional arguments passed qsys constructor","code":""},{"path":"/reference/workers.html","id":"value","dir":"Reference","previous_headings":"","what":"Value","title":"Creates a pool of workers — workers","text":"instance QSys class","code":""},{"path":"/reference/wrap_error.html","id":null,"dir":"Reference","previous_headings":"","what":"Wraps an error in a condition object — wrap_error","title":"Wraps an error in a condition object — wrap_error","text":"Wraps error condition object","code":""},{"path":"/reference/wrap_error.html","id":"ref-usage","dir":"Reference","previous_headings":"","what":"Usage","title":"Wraps an error in a condition object — wrap_error","text":"","code":"wrap_error(call)"},{"path":[]},{"path":"/news/index.html","id":"features-0-9-0","dir":"Changelog","previous_headings":"","what":"Features","title":"clustermq 0.9.0","text":"Reuse common data now supported (#154) Jobs now error instead stalling upon unexpected worker disconnect (#150) Workers now error can establish connection within time limit Error n_jobs max_calls_worker provide insufficient call slots (#258) Request 1 GB default SGE template (#298) @nickholway Error warning summary now orders index severity (#304) call can multiple warnings forwarded, last","code":""},{"path":"/news/index.html","id":"bugfix-0-9-0","dir":"Changelog","previous_headings":"","what":"Bugfix","title":"clustermq 0.9.0","text":"Fix bug max memory reporting gc() may different column (#240) Fix passing numerical job_id qdel PBS (#265) job port/id pool now used properly upon binding failure (#270) @luwidmer Common data size warning now displayed exceeding limits (#287)","code":""},{"path":"/news/index.html","id":"under-the-hood-0-9-0","dir":"Changelog","previous_headings":"","what":"Under the hood","title":"clustermq 0.9.0","text":"Complete rewrite worker API longer depend purrr package","code":""},{"path":"/news/index.html","id":"clustermq-0895","dir":"Changelog","previous_headings":"","what":"clustermq 0.8.95","title":"clustermq 0.8.95","text":"CRAN release: 2020-07-01 now using ZeroMQ via Rcpp preparation v0.9 (#151) New multiprocess backend via callr instead forking (#142, #197) Sending data sockets now blocking avoid excess memory usage (#161) multicore, multiprocess schedulers now support logging (#169) New option clustermq.host can specify host IP network interface name (#170) Template filling now raise error missing keys (#174, #198) Workers failing large common data improved (fixed?) (#146, #179, #191) Local connections now routed via 127.0.0.1 instead localhost (#192) Submit messages different local, multicore HPC (#196) Functions exported foreach now environment stripped (#200) Deprecation log_worker=T/F argument rescinded","code":""},{"path":"/news/index.html","id":"clustermq-089","dir":"Changelog","previous_headings":"","what":"clustermq 0.8.9","title":"clustermq 0.8.9","text":"CRAN release: 2020-02-29 New option clustermq.ssh.timeout SSH proxy startup (#157) @brendanf New option clustermq.worker.timeout delay worker shutdown (#188) Fixed PBS/Torque docs, template cleanup (#184, #186) @mstr3336 Warning common data large, set clustermq.data.warning (#189)","code":""},{"path":"/news/index.html","id":"clustermq-088","dir":"Changelog","previous_headings":"","what":"clustermq 0.8.8","title":"clustermq 0.8.8","text":"CRAN release: 2019-06-05 Q, Q_rows new arguments verbose (#111) pkgs (#144) foreach backend now uses dedicated API possible (#143, #144) Number size objects common calls now work properly Templates filled internally longer depend infuser package","code":""},{"path":"/news/index.html","id":"clustermq-087","dir":"Changelog","previous_headings":"","what":"clustermq 0.8.7","title":"clustermq 0.8.7","text":"CRAN release: 2019-04-15 Q now max_calls_worker argument avoid walltime (#110) Submission messages now list size common data (drake#800) default templates now optional cores per job field (#123) foreach now treats .export (#124) .combine (#126) correctly New option clustermq.error.timeout wait clean shutdown (#134) SSH command now specified via template file (#122) SSH now forward errors local process (#135) Wiki deprecated, use https://mschubert.github.io/clustermq/ instead","code":""},{"path":"/news/index.html","id":"clustermq-086","dir":"Changelog","previous_headings":"","what":"clustermq 0.8.6","title":"clustermq 0.8.6","text":"CRAN release: 2019-02-22 Progress bar now shown workers start (#107) Socket connections now authenticated using session password (#125) Marked internal functions @keywords internal Added vignettes User Guide Technical Documentation","code":""},{"path":"/news/index.html","id":"clustermq-085","dir":"Changelog","previous_headings":"","what":"clustermq 0.8.5","title":"clustermq 0.8.5","text":"CRAN release: 2018-09-29 Added experimental support parallel foreach backend (#83) Moved templates package inst/ directory (#85) Added send_call worker evaluate arbitrary expressions (drake#501; #86) Option clustermq.scheduler now respected set package load (#88) System interrupts now handled correctly (rzmq#44; #73, #93, #97) Number workers running/total now shown progress bar (#98) Unqualified (short) host names now resolved default (#104)","code":""},{"path":"/news/index.html","id":"clustermq-084","dir":"Changelog","previous_headings":"","what":"clustermq 0.8.4","title":"clustermq 0.8.4","text":"CRAN release: 2018-04-22 Fix error qsys$reusable using n_jobs=0/local processing (#75) Scheduler-specific templates deprecated. Use clustermq.template instead Allow option clustermq.defaults fill default template values (#71) Errors worker processing now shut cleanly (#67) Progress bar now shows estimated time remaining (#66) Progress bar now also shown processing locally Memory summary now adds estimated memory R session (#69)","code":""},{"path":"/news/index.html","id":"clustermq-083","dir":"Changelog","previous_headings":"","what":"clustermq 0.8.3","title":"clustermq 0.8.3","text":"CRAN release: 2018-01-21 Support rettype function calls return type known (#59) Reduce memory requirements processing results receive Fix bug cleanup, log_worker flag working SGE/SLURM","code":""},{"path":"/news/index.html","id":"clustermq-082","dir":"Changelog","previous_headings":"","what":"clustermq 0.8.2","title":"clustermq 0.8.2","text":"CRAN release: 2017-11-30 Fix bug never-started jobs cleaned Fix bug tests leave processes port binding fails (#60) Multicore longer prints worker debug messages (#61)","code":""},{"path":"/news/index.html","id":"clustermq-081","dir":"Changelog","previous_headings":"","what":"clustermq 0.8.1","title":"clustermq 0.8.1","text":"CRAN release: 2017-11-27 Fix performance issues high number function calls (#56) Fix bug multicore workers shut properly (#58) Fix default templates SGE, LSF SLURM (misplaced quote)","code":""},{"path":"/news/index.html","id":"clustermq-080","dir":"Changelog","previous_headings":"","what":"clustermq 0.8.0","title":"clustermq 0.8.0","text":"CRAN release: 2017-11-11","code":""},{"path":"/news/index.html","id":"features-0-8-0","dir":"Changelog","previous_headings":"","what":"Features","title":"clustermq 0.8.0","text":"Templates changed: clustermq:::worker now takes master argument Creating workers now separated Q, enabling worker reuse (#45) Objects function environment must now exported explicitly (#47) Added multicore qsys using parallel package (#49) New function Q_rows using data.frame rows iterated arguments (#43) Job summary now report max memory reported gc (#18)","code":""},{"path":"/news/index.html","id":"bugfix-0-8-0","dir":"Changelog","previous_headings":"","what":"Bugfix","title":"clustermq 0.8.0","text":"Fix bug copies common_data collected gc slowly (#19)","code":""},{"path":"/news/index.html","id":"under-the-hood-0-8-0","dir":"Changelog","previous_headings":"","what":"Under the hood","title":"clustermq 0.8.0","text":"Messages master now processed threads (#42) Jobs now submitted array possible","code":""},{"path":"/news/index.html","id":"clustermq-070","dir":"Changelog","previous_headings":"","what":"clustermq 0.7.0","title":"clustermq 0.7.0","text":"CRAN release: 2017-08-28 Initial release CRAN","code":""}] diff --git a/sitemap.xml b/sitemap.xml index cb564c1..1f480f7 100644 --- a/sitemap.xml +++ b/sitemap.xml @@ -39,6 +39,9 @@ /reference/MULTIPROCESS.html + + /reference/Pool.html + /reference/Q.html @@ -57,12 +60,6 @@ /reference/SSH.html - - /reference/ZeroMQ.html - - - /reference/bind_avail.html - /reference/check_args.html @@ -70,7 +67,7 @@ /reference/chunk.html - /reference/clustermq.html + /reference/clustermq-package.html /reference/cmq_foreach.html @@ -94,7 +91,7 @@ /reference/master.html - /reference/purrr_lookup.html + /reference/msg_fmt.html /reference/register_dopar_cmq.html @@ -117,4 +114,7 @@ /reference/workers.html + + /reference/wrap_error.html +