string to numeric array

Question

From one program I generate bunch of data and it is stored in a file. An example of the file's contents is

[[1, 2, 3], [4, 5, 6]]

As you can see, the data has the exact form of an array. Later in another program I want to read the data and use it. I am using

text_file = open('DataFile.txt')
lines = text_file.readlines() #We have only 1 line but this doesn't matter

The variable lines is an array of 1 element which is the string [[1, 2, 3], [4, 5, 6]]. I want this string to be again a numeric array. Just with the same delimiters etc. How can I do this?

Answer 1

You can use ast.literal_eval:

>>> from ast import literal_eval
>>> mystr = '[[1, 2, 3], [4, 5, 6]]'
>>> x = literal_eval(mystr)
>>> x
[[1, 2, 3], [4, 5, 6]]
>>> type(x)
<type 'list'>
>>>

Answer 2

The ast module provides a literal_eval(repr) function that safely evaluates any python literal:

>>> import ast
>>> ast.literal_eval("[[1,2,3,4], [5,6,7,8]]")
[[1, 2, 3, 4], [5, 6, 7, 8]]
>>> ast.literal_eval("19e6")
19000000.0

Alternatively, you could use json.loads:

>>> json.loads("[[1,2,3,4,5], [6,7,8,9]]")
[[1, 2, 3, 4, 5], [6, 7, 8, 9]]

The json solution is quite faster to literal_eval:

In [1]: import ast, json

In [2]: %timeit ast.literal_eval("[[1,2,3,4], [5,6,7,8]]")
10000 loops, best of 3: 33.5 us per loop

In [3]: %timeit json.loads("[[1,2,3,4], [5,6,7,8]]")
100000 loops, best of 3: 4.16 us per loop

In [4]: 33.5 / 4.16
Out[4]: 8.052884615384615
In [5]: big_list_text = '[{}]'.format(','.join(["[1,2,3,4]"] * 1000))

In [6]: %timeit ast.literal_eval(big_list_text)
100 loops, best of 3: 14.6 ms per loop

In [7]: %timeit json.loads(big_list_text)
1000 loops, best of 3: 485 us per loop

In [8]: 14600 / 485
Out[8]: 30.103092783505154

As you can see json.loads is significantly faster than ast.literal_eval, although they parse different things which only happen to be the same when dealing with simple list literals.