It would be really nice if the save and load functions supported parallel compression from the pbzip2 and pigz libraries. Compressing in parallel can dramatically speed up a workflow.
Please supply evidence of a real problem in which this is the case.
(In reply to comment #1)
> Please supply evidence of a real problem in which this is the case.
Using bzip2 I get this timing:
> print(object.size(a), units='auto')
> system.time(save(a, file='car.RData', compress='bzip2'))
user system elapsed
165.892 2.112 169.956
On my ec2 instance with a pretty fast disk drive I get this when using pbzip2
> save(a, file='car3.RData')
> system('pbzip2 car3.RData')
Time difference of 49.23157 secs
I believe the pbzip2 workflow includes an additional write to disk. So it would be even faster if we could do something like
save(a, file='car.RData', compress='pbzip2')