How to read line-delimited JSON from large file (line by line)

Question

I'm trying to load a large file (2GB in size) filled with JSON strings, delimited by newlines. Ex:

{     "key11": value11,     "key12": value12, } {     "key21": value21,     "key22": value22, } … 

The way I'm importing it now is:

content = open(file_path, "r").read()  j_content = json.loads("[" + content.replace("}\n{", "},\n{") + "]") 

Which seems like a hack (adding commas between each JSON string and also a beginning and ending square bracket to make it a proper list).

Is there a better way to specify the JSON delimiter (newline \n instead of comma ,)?

Also, Python can't seem to properly allocate memory for an object built from 2GB of data, is there a way to construct each JSON object as I'm reading the file line by line? Thanks!

Answer 1

Just read each line and construct a json object at this time:

with open(file_path) as f:     for line in f:         j_content = json.loads(line) 

This way, you load proper complete json object (provided there is no \n in a json value somewhere or in the middle of your json object) and you avoid memory issue as each object is created when needed.

There is also this answer.:

https://stackoverflow.com/a/7795029/671543

Answer 2

contents = open(file_path, "r").read()  data = [json.loads(str(item)) for item in contents.strip().split('\n')]