How to save large output sufficiently fast in text or any other format?

Question

My question is: how to save the output i.e., mydata

 mydata=array(sample(100),dim=c(2,100,4000))

reasonably fast?

I used the reshape2 package as suggested here.

 melt(mydata)

and

 write.table(mydata,file="data_1")

But it is taking more than one hour to save the data into the file. I am looking for any other faster ways to do the job.

Answer 1

I strongly suggest to refer to this great post, that surely helps in make issues clear about file saving.

Anyway, saveRDS could be the most adequate for you. The difference more relevant, in this case, is that save can save many objects to a file in a single call, whilst saveRDS, being a lower-level function, works with a single object at a time.

save and load allow you to save a named R object to a file or other connection and restore that object again. But, when loaded, the named object is restored to the current environment with the same name it had when saved.

saveRDS and loadRDS, instead, allow to save a single R object to a connection (typically a file) and to restore the object, possibly with a different name. The low level operability makes RDS functions more efficient, probably, for your case.

Answer 2

Read the help text for saveRDS using ?saveRDS. This will probably be the best way for you to save and load large dataframes.

saveRDS(yourdata, file = "yourdata.Rda")