How to read a large text file line-by-line and append this stream to a file line-by-line in Ruby?

Question

Let's say I want to combine several massive files into one and then uniq! the one (THAT alone might take a hot second)

It's my understanding that File.readlines() loads ALL the lines into memory. Is there a way to read it line by line, sort of like how node.js pipe() system works?

Answer 1

One of the great things about Ruby is that you can do file IO in a block:

File.open("test.txt", "r").each_line do |row|
  puts row
end               # file closed here

so things get cleaned up automatically. Maybe it doesn't matter on a little script but it's always nice to know you can get it for free.

Answer 2

you aren't operating on the entire file contents at once, and you don't need to store the entirety of each line either if you use readline.

file = File.open("sample.txt", 'r')
while !file.eof?
   line = file.readline
   puts line
end