Feature request: Evaluation of expression on each worker

[stephensrmmartin]

doRedis is really cool, but I would love a straightforward way to execute expressions on each worker.
Why you ask?

Rstan/Stan compiles templates down to C++, and ultimately to a dso. My workstation has a different cpu and GCC version than our cluster.

What I tend to do, is build the model on the cluster, and on each worker load it in.
E.g., clusterEvalQ(cl, sMod <- readRDS('mystanmodel.rds'))
or clusterEvalQ(cl, sMod <- stan_model('mystanmodel.stan'))
foreach()'s export will not work for this problem, because it will try to export the local sMod, which is incompatible with the cluster's runtime and likely the CPU itself. In fact, I have to use .noexport='sMod', otherwise it will not be able to find and load the .dso file.

To my knowledge, there is no alternative.
So ideally, there would be a doRedis function that doesn't require foreach, and simply sends the same job to every worker to execute in its session. Then foreach() could be run without issue.
If there is some other method that I am missing, I'd love to hear it.

[bwlewis]

Very sorry for the long latency in responding. Check out the vignette section 5.2. You can explicitly export a function called "worker.init()" with no arguments. That will be run exactly once per job (at the start) per worker. It gets run in the global environment on the worker. I think that will do what you want.

…

On 2/5/19, Stephen Martin ***@***.***> wrote: doRedis is really cool, *but* I would love a straightforward way to execute expressions on each worker. Why you ask? Rstan/Stan compiles templates down to C++, and ultimately to a dso. My workstation has a different cpu and GCC version than our cluster. What I tend to do, is build the model on the cluster, and on each worker load it in. E.g., clusterEvalQ(cl, sMod <- readRDS('mystanmodel.rds')) or clusterEvalQ(cl, sMod <- stan_model('mystanmodel.stan')) foreach()'s export will not work for this problem, because it will try to export the local sMod, which is incompatible with the cluster's runtime and likely the CPU itself. In fact, I have to use .noexport='sMod', otherwise it will not be able to find and load the .dso file. To my knowledge, there is no alternative. So ideally, there would be a doRedis function that doesn't require foreach, and simply sends the same job to every worker to execute in its session. Then foreach() could be run without issue. If there is some other method that I am missing, I'd love to hear it. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: #51