Python regex for module-level constants

Question

I have looked back and forward for a possible solution to my issue, but I guess my google-fu is very poor today. Not to mention my knowledge of regular expressions, which is close to zero.

I am trying to retrieve the definition of some "constants" (i.e., module-level global variables) from a series of Python files using text processing only. Basically, I read the whole text of a Python file and then I apply a regex to the text to find out where/what these "constants" are.

As an example, I can have in my Python file something like this:

CONSTANT_ONE   = 0    # standard
CONSTANT_RIGHT = 1    # rotation of 90 on the right
CONSTANT_LEFT  = 2    # rotation of 90 on the left

And I found out that this regex works OK to extract the variable names:

re.compile('^(\w+)[ \t]*=', re.M)

However, it badly fails with declarations like these:

NAME1, NAME2 = 0, None
CONST_1, CONST_2, CONST_3 = range(3)

Is there any way in which I can modify my regex to handle both cases? My apologies if this is a very basic question but I am really no expert of regex...

Thank you in advance.

Andrea.

Answer 1

Don't use regular expressions, use Python's parser instead. It is much simpler:

class TargetExtractor(ast.NodeVisitor):
    def visit_Name(self, node):
        if isinstance(node.ctx, ast.Store):
            print node.id
    def visit_FunctionDef(self, node):
        pass
    def visit_ClassDef(self, node):
        pass

TargetExtractor().visit(ast.parse("a, b = 2, 3; c = d"))

prints

a
b
c

Instead of the string, you can also pass the whole contents of a file to ast.parse().

Answer 2

I don't think there is an easy way to achieve this using regular expressions. It's possible to compile the Python modules and walk down the AST to find variable definitions. See the documentation at http://docs.python.org/library/functions.html#compile and http://docs.python.org/library/ast.html#module-ast.

EDIT: My current program, using the idea from @Sven.

#!/usr/bin/env python

import ast
import sys

# example assignment.
a, b=5, 9

class MyNodeVisitor(ast.NodeVisitor):
        """
        Visit nodes in AST. Idea from @Sven.
        """
        def visit_Name(self, node):
                if isinstance(node.ctx, ast.Store):
                        print "Assigning name '%s' on line %d" % ( \
                                node.id, node.lineno)
        def visit_FunctionDef(self, node):
                pass
        def visit_ClassDef(self, node):
                pass

def printAssignments(name):
        """
        Read Python file and print a list of variable assignments.
        """
        # read file.
        f=open(name, 'rU')
        data=f.readlines()
        f.close()

        # create AST.
        t=ast.parse("".join(data), filename=name, mode='exec')

        # find assignments.
        MyNodeVisitor().visit(t)

        # walk nodes.
        # for node in ast.walk(t):
        #       if isinstance(node, ast.Name) \
        #          and isinstance(node.ctx, ast.Store) \
        #          and node.col_offset == 0:
        #               print "Assigning name '%s' on line %d" % (node.id, node.lineno)

# print all assignments from files.
for name in sys.argv[1:]:
        print "=== %s ===" % name
        printAssignments(name)
        print "====%s====" % ('='*len(name), )