Errno::EINVAL Invalid argument @ io_fread

Question

I'm attempting to download a ~2GB file and write it to a file locally but I'm running into this issue:

Here's the applicable code:

  File.open(local_file, "wb") do |tempfile|
    puts "Downloading the backup..."

    pbar = nil
    open(backup_url,
         :read_timeout => nil,
         :content_length_proc => lambda do |content_length|
           if content_length&.positive?
             pbar = ProgressBar.create(:total => content_length)
           end
         end,
         :progress_proc => ->(size) { pbar&.progress = size }) do |retrieved|
          begin
            tempfile.binmode
            tempfile << retrieved.read
            tempfile.close
          rescue Exception => e
            binding.pry
          end
    end

Answer 1

Read your file in chunks.

The line causing the issue is here:

tempfile << retrieved.read

This reads the entire contents into memory before writing it to the tempfile. If the content is small, this isn't a big deal, but if this content is quite large (how large depends on the system, configuration, OS and available resources), this can cause an Errno::EINVAL error, like Invalid argument @ io_fread and Invalid argument @ io_write.

To work around this, read the content in chunks and write each chunk to the tempfile. Something like this:

tempfile.write( retrieved.read( 1024 ) ) until retrieved.eof?

This will get chunks of 1024 bytes and write each chunk to the tempfile until retrieved reaches the end of the file (i.e. .eof?).

If retrieved.read doesn't take a size parameter, you may need to convert retrieved into a StringIO, like this:

retrievedIO = StringIO.new( retrieved )
tempfile.write( retrievedIO.read( 1024 ) ) until retrievedIO.eof?