Bug 16793 - Make a way to get amount of available memory on Linux
Summary: Make a way to get amount of available memory on Linux
Status: UNCONFIRMED
Alias: None
Product: R
Classification: Unclassified
Component: System-specific (show other bugs)
Version: R 3.2.4
Hardware: x86_64/x64/amd64 (64-bit) Linux
: P2 enhancement
Assignee: R-core
URL:
Depends on:
Blocks:
 
Reported: 2016-04-02 11:51 UTC by Adam Ryczkowski
Modified: 2016-04-02 19:06 UTC (History)
1 user (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Adam Ryczkowski 2016-04-02 11:51:57 UTC
The usual trick with `system(<shell command that gets the free memory>)` is far from perfect.

Spawning the subshell has heavy memory footprint in itself, and I've seen it many times to fail while still there were many MB of memory available.

It is not compatible with per-process memory constraints assigned by kernel API. 


Why it is not a minor inconvenience?
------------------------------------

Usually there are opportunities to paralallize several computations (using the `mcparallel` from the `parallel` library), but this can fail due to insufficient memory. It would be way more efficient to check the amount of available memory beforehand, but is is currently not implemented on R. The usual trick with `system(<shell command that gets the free memory>)` usually is insufficient:

When one tries to work with a bigger data on linux there is a real danger of locking the workstation up with excess paging when R happily allocates too much memory. The natural solution is to use kernel API to impose memory constraints upon R, like the package https://github.com/krlmlr/ulimit
Comment 1 Dirk Eddelbuettel 2016-04-02 19:06:56 UTC
Disagree.  These things are necessarily OS-dependent, and rely on reasonable and responsible system administration.  See eg http://unix.stackexchange.com/questions/44985/limit-memory-usage-for-a-single-linux-process

The R Installation and Administration manual already addresses this in Section 5:
"You should ensure that the shell has set adequate resource limits: R expects a stack size of at least 8MB and to be able to open at least 256 file descriptors. (Any modern OS should have default limits at least as large as these, but apparently NetBSD may not. Use the shell command ulimit (sh/bash) or limit (csh/tcsh) to check.)"