File editing in python

Question

I wrote a little python program as a personal utility to help me with some refactoring. It's similar to unix replace, except it supports regular expressions and operates on all files in a directory and (optionally) all subdirectories.

The problem is I'm not replacing in place. I'm opening files, passing the contents into memory, and then overwriting the file, like so:

file = open('path/to/file','r')
in_string = file.read()
file.close()
# ...
#Processing logic
# ...
file = open('path/to/file','w')
file.write(out_string)
file.close()

Besides the obvious performance/memory concerns, which are legitimate, but not so much a problem for my use, there's another drawback to this method. SVN freaks out. I can do some copy and paste workarounds after the fact to fix the checksum error that svn throws on a commit, but it makes the utility pointless.

Is there a better way to do this? I'm guessing that if I were editing the file in place there wouldn't be any sort of problem. How do I go about that?

Answer 1

I suspect the problem is that you are in fact editing wrong files. Subversion should never raise any errors about check sums when you are just modifying your tracked files -- independently of how you are modifying them.

Maybe you are accidentally editing files in the .svn directory? In .svn/text-base, Subversion stores copies of your files using the same name plus the extension .svn-base, make sure that you are not editing those!

Answer 2

What do you mean by "SVN freaks out"?

Anyway, the way vi/emacs/etc works is as follows:

f = open("/path/to/.file.tmp", "w")
f.write(out_string)
f.close()
os.rename("/path/to/.file.tmp", "/path/to/file")

(ok, there's actually an "fsync" in there... But I don't know off hand how to do that in Python)

The reason it does that copy is to ensure that, if the system dies half way through writing the new file, the old one is still there... And the 'rename' operation is defined as being atomic, so it will either work (you get 100% of the new file) or not work (you get 100% of the old file) -- you'll never be left with a half-file.