Save large numeric output to file natively in Julia 1.0.0

Question

I am trying to run a program in hpc-cluster. Unfortunately, I am unable to install external packages (e.g., JLD2) on the cluster. This is is a temporary problem, and should get fixed.

I don't want to wait around all that time and I am wondering if there is any way to save large output (2-3 GB) in Julia without external dependencies. Most of the output is matrix of numbers. I was using JLD2 previously that stores data in HDF5 format.

Bonus question: Is there a workaround to this using shell commands, like use pipe to get the output and use awk//grep to save data? (something like julia -p 12 main.jl | echo "file").

Answer 1

You could try the Serialization standard library.

To work with multiple variables, you can just store them sequentially:

x = rand(10)
y = "foo"

using Serialization
# write to file
open("data.out","w") do f
    serialize(f, x)
    serialize(f, y)
end

# load from file
open("data.out") do f
    global x2, y2
    x2 = deserialize(f)
    y2 = deserialize(f)
end

or you could put them in a Dict, and just store that.