remoter with BiocParallel

[inquil]

Hi again,

each time I use a parallel implementation of lapply or mapply using the Bioconductor package BiocParallel (bplapply, bpmapply) remoter at the server side is killed, producing the message "R_zmq_msg_recv errno: 4 strerror: Interrupted system call"
It seems a problem with the BiocParallel package though. Same command directly executed at the server manually does not produce any problem.
Regards,
Carlos

[wrathematics]

I just did a quick test with bplapply(1:10, sqrt) and things seemed ok. Do you have some example code you could share?

[inquil]

Hi,

bpparam <- MulticoreParam(workers = 10)
system.time ( lista_views <- bpmapply(coverageabove,genome_file,cds.df$seqnames,as.numeric(cds.df$start),as.numeric(cds.df$end),cds.df$strand, USE.NAMES = FALSE, BPPARAM=bpparam) )

Best regards,

Carlos

[wrathematics]

Interesting. So I've found a machine where this fails. However, it doesn't fail when I use a different backend, like SnowParam(). So could you try using a different backend from the client, like:

BiocParallel::bplapply(1:5, sqrt, BPPARAM=SnowParam())

I'd also be curious to know if parallel::mclapply(1:5, sqrt) crashes things, because it doesn't for me (on the machine where bplapply() things break). This sort of makes me think it's more a problem with BiocParallel than remoter, but I'm not sure.

Maybe someone from the BiocParallel project has some ideas. Pinging @mtmorgan.

[inquil]

HI,
it works well with
SnowParam(workers=5,type="SOCK")
but not with
type="FORK"
As you said, It seems that all open sockets are closed when "FORK" is used, including those of remoter.
Best regards,
Carlos

[mtmorgan]

A non-BiocParallel example that kills the server for me is

remoter> library(parallel)
remoter> cl = makeForkCluster(2)
remoter> stopCluster(cl)

I think it is from parallel:::mcexit(), which calls _exit(), which in turn generates a SIGCHLD signal, but I'm not really sure.

[wrathematics]

Thanks for the quick response! Unfortunately(/fortunately?) that doesn't kill the server for me; does it for you?

The only way so far I've been able to recreate the server crashing is on one machine with bplapply() with the fork backend.

[inquil]

Hi,
this works well:
parallel::mclapply(1:5, sqrt)

but last non-BiocParallel example from @mtmorgan kills the server:

[2016-09-15 08:05:00]: *** Launching UNSECURE server ***
[2016-09-15 08:05:18]: client connected
[2016-09-15 08:05:38]: RMSG: library(parallel)
[2016-09-15 08:06:04]: RMSG: cl = makeForkCluster(2)
[2016-09-15 08:06:14]: RMSG: stopCluster(cl)
R_zmq_msg_recv errno: 4 strerror: Interrupted system call
Error in unserialize(rmsg) : read error
>

Regards,
Carlos

[wrathematics]

This is so strange; I can't replicate it. I really have no idea what's going on.

[zhuyuecai]

Just curious how this issue is going on.
Since I am going to implement a centralize R server based on remoter in the lab, and a lot of scripts from bioconductor will be run on it.