Created attachment 1124 [details]
It seems there is a small bug in memory.limit on the win64 builds. It prevents to set the memory limit to values greater than 4095 during an R session:
In memory.size(size) : cannot decrease memory limit: ignored
R version 2.11.1 Patched (2010-08-17 r52771) / 2.12.0 trunk svn52852
Platform: x86_64-pc-mingw32/x64 (64-bit)
I guess the type of the variable newmax in the associated C function
SEXP do_memsize(SEXP call, SEXP op, SEXP args, SEXP rho)
is not yet adapted for 64bit compatibility? Changing it to R_size_t fixes the issue for me (see patch).
Thanks for the patch, I'll put it in.