Parse HTML style text annotations to a list of dictionaries

Question

Currently I have the following problem:

Given a string

"<a>annotated <b>piece</b></a> of <c>text</c>" 

construct a list such that the result is

[{"annotated": ["a"]}, {"piece": ["a", "b"]}, {"of": []}, {"text": ["c"]}].

My previous attempt looked something like

open_tag = '<[a-z0-9_]+>'
close_tag = '<\/[a-z0-9_]+>'
tag_def = "(" + open_tag + "|" + close_tag + ")"

def tokenize(str):
    """
    Takes a string and converts it to a list of words or tokens
    For example "<a>foo</a>, of" -> ['<a>', 'foo', '</a>', ',' 'of']
    """
    tokens_by_tag = re.split(tag_def, str)
    def tokenize(token):
        if not re.match(tag_def, token):
            return word_tokenize(token)
        else:
            return [token]
    return list(chain.from_iterable([tokenize(token) for token in tokens_by_tag]))

def annotations(tokens):
    """
    Process tokens into a list with {word : [tokens]} items
    """
    mapping = []
    curr = []
    for token in tokens:
        if re.match(open_tag, token):
            curr.append(re.match('<([a-z0-9_]+)>',token).group(1))
        elif re.match(close_tag, token):
            tag = re.match('<\/([a-z0-9_]+)>',token).group(1)
            try:
                curr.remove(tag)
            except ValueError:
                pass
        else:
            mapping.append({token: list(curr)})
    return mapping

Unfortunately this has a flaw since (n=54) resolves to {"n=54" : []} but (n=<n>52</n>) to [{"n=": []}, {52: ["n"]}] thus the length of the two lists differ, making it impossible to merge two different ones later on.

Is there a good strategy for doing parsing HTML/SGML style annotations in a way that two differently annotated (but otherwise identical) strings yield a list of equal size?

Note that I'm well aware that regexps are not suitable for this type of parsing, but also not the problem in this case.

EDIT Fixed a mistake in the example

Answer 1

Your xml data (or html) is not well formed.

Assuming an input file with following xml data well formed:

<root><a>annotated <b>piece</b></a> of <c>text</c></root>

you can use a sax parser to append and pop tags at start and end element events:

from xml.sax import make_parser
from xml.sax.handler import ContentHandler
import sys 

class Xml2PseudoJson(ContentHandler):

    def __init__(self):
        self.tags = []
        self.chars = []
        self.json = []

    def startElement(self, tag, attrs):
        d = {''.join(self.chars): self.tags[:]}
        self.json.append(d)
        self.tags.append(tag)
        self.chars = []

    def endElement(self, tag):
        d = {''.join(self.chars): self.tags[:]}
        self.chars = []
        self.tags.pop()
        self.json.append(d)

    def characters(self, content):
        self.chars.append(content)

    def endDocument(self):
        print(list(filter(lambda x: '' not in x, self.json)))

parser = make_parser()
handler = Xml2PseudoJson()
parser.setContentHandler(handler)
parser.parse(open(sys.argv[1]))

Run it like:

python3 script.py xmlfile

That yields:

[
    {'annotated ': ['root', 'a']}, 
    {'piece': ['root', 'a', 'b']}, 
    {' of ': ['root']}, 
    {'text': ['root', 'c']}
]