Bug 17343 - parallel::mclapply does not recognize/signal if child process is killed
Summary: parallel::mclapply does not recognize/signal if child process is killed
Status: NEW
Alias: None
Product: R
Classification: Unclassified
Component: Add-ons (show other bugs)
Version: R 3.4.1
Hardware: x86_64/x64/amd64 (64-bit) Linux
: P5 enhancement
Assignee: R-core
URL:
Depends on:
Blocks:
 
Reported: 2017-09-22 13:42 UTC by Andreas Kersting
Modified: 2017-10-08 10:49 UTC (History)
1 user (show)

See Also:


Attachments
make mclapply return a fork-error if cores did not return (2.64 KB, patch)
2017-10-08 10:47 UTC, Andreas Kersting
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Andreas Kersting 2017-09-22 13:42:45 UTC
On Linux, run

###
x <- parallel::mclapply(1:2, function(i) {
  Sys.sleep(30L)
  i
}, mc.cores = 2L)

print(x)
###

and kill one of the two forked process. 

This will print

[[1]]
NULL

[[2]]
[1] 2

if the first of the two processes is killed and 

[[1]]
[1] 1

if the second one is killed.

I expect parallel::mclapply() to throw an error if one or more of the child processes fail due to a fatal error or get killed for other reasons, e.g. by oom-killer.
Comment 1 Tomas Kalibera 2017-10-06 15:57:07 UTC
I have partially addressed this in 73476: now one gets a NULL in the results no matter which of the spawned processes crashes or gets killed. So when the function ('FUN') called via mclapply only returns non-NULLs, one can detect errors by checking for NULLs in the result list (and for objects of class "try-error" for user/R errors).

The current behavior (that mclapply does not throw an error itself) is good in that one can get at least partial results.
Comment 2 Andreas Kersting 2017-10-08 10:47:42 UTC
Created attachment 2300 [details]
make mclapply return a fork-error if cores did not return
Comment 3 Andreas Kersting 2017-10-08 10:49:14 UTC
Thanks for looking into this issue, Tomas.

For a non-fatal error I totally agree that mclapply should not throw an error, for a fatal one I am not so sure: If I ran the same code without mclapply it would have crashed my main R process and hence mclapply is kind of acting as a try-catch for fatal errors, which is something you (I) would not expect.

Anyway, attached you can find a patch proposal to make mclapply return "fork-error"-objects (instead of NULL) if a core did not return. This might be a good compromise and is similar to "try-error"-objects being returned for errors in user code.

Of course, the behavior should be independent of mc.preschedule. I only implemented it for the "mc.preschedule = TRUE"-case, as I am not sure if this is a viable and clean solution. Someone knowing the code better (you?) might find a more appropriate way to implement the original idea.