python: how to create persistent in-memory structure for debugging

Question

[Python 3.1]

My program takes a long time to run just because of the pickle.load method on a huge data structure. This makes debugging very annoying and time-consuming: every time I make a small change, I need to wait for a few minutes to see if the regression tests passed.

I would like replace pickle with an in-memory data structure.

I thought of starting a python program in one process, and connecting to it from another; but I am afraid the inter-process communication overhead will be huge.

Perhaps I could run a python function from the interpreter to load the structure in memory. Then as I modify the rest of the program, I can run it many times (without exiting the interpreter in between). This seems like it would work, but I'm not sure if I will suffer any overhead or other problems.

Answer 1

You can use mmap to open a view on the same file in multiple processes, with access at almost the speed of memory once the file is loaded.