Complex parsing of a string in Python

Question

I want to parse a string with a format like this:

[{text1}]{quantity}[{text2}]

This rule means that at the beginning there is some text that can optionally be there or not, followed by a {quantity} whose syntax I describe just below, followed by more optional text.

The {quantity} can take a variety of forms, with {n} being any positive integer

{n}
{n}PCS
{n}PC
{n}PCS.
{n}PC.
Lot of {n}

Also, it should accept this additional rule:

{n} {text2} 

In this rule, {n} is followed by a space then {text2}

In the cases where PC or PCS appears

• it may or may not be followed by a dot
• case insensitive
• a space can optionally appear between {n} and PCS
• The following are all stripped: PC or PCS, the optional dot, and the optional space

The desired output is normalized to two variables:

• {n} as an integer
• [{text1}] [{text2}], that is, first {text1} (if present), then a space, then {text2} (if present), concatenated to one string. A space to separate the text pieces is only used if there are two of them.

If the {quantity} includes anything besides a positive integer, {n} consists only of the the integer, and the rest of {quantity} (e.g. " PCS.") is stripped from both {n} and the resultant text string.

In the text parts, more integers could appear. Any other than the {quantity} found should be regarded as just part of the text, not interpreted as another quantity.

I am a former C/C++ programmer. If I had to solve this with those languages, I would probably use rules in lex and yacc, or else I would have to write a lot of nasty code to hand-parse it.

I would like to learn a clean approach for coding this efficiently in Python, probably using rules in some form to easily support more cases. I think I could use lex and yacc with Python, but I wonder if there is an easier way. I'm a Python newbie; I don't even know where to start with this.

I am not asking anyone to write code for a complete solution, rather, I need an approach or two, and perhaps some sample code showing part of how to do it.

Answer 1

Pyparsing let's you build up a parser by stitching together smaller parsers using '+' and '|' operators (among others). You can also attach names to the individual elements in the parser, to make it easier to get at the values afterward.

from pyparsing import (pyparsing_common, CaselessKeyword, Optional, ungroup, restOfLine, 
    oneOf, SkipTo)

int_qty = pyparsing_common.integer

# compose an expression for the quantity, in its various forms
"""
{n}
{n}PCS
{n}PC
{n}PCS.
{n}PC.
Lot of {n}
"""
LOT = CaselessKeyword("lot")
OF = CaselessKeyword("of")
pieces = oneOf("PC PCS PC. PCS.", caseless=True)
qty_expr = Optional(LOT + OF).suppress() + int_qty("qty") + Optional(pieces).suppress()

# compose expression for entire line
line_expr = SkipTo(qty_expr)("text1") + qty_expr + restOfLine("text2")

tests = """
    Send me 1000 widgets pronto!
    Deliver a Lot of 50 barrels of maple syrup by Monday, June 10.
    My shipment was short by 25 pcs.
    """

line_expr.runTests(tests)

Prints:

Send me 1000 widgets pronto!
['Send me', 1000, ' widgets pronto!']
- qty: 1000
- text1: ['Send me']
- text2:  widgets pronto!


Deliver a Lot of 50 barrels of maple syrup by Monday, June 10.
['Deliver a ', 50, ' barrels of maple syrup by Monday, June 10.']
- qty: 50
- text1: ['Deliver a ']
- text2:  barrels of maple syrup by Monday, June 10.


My shipment was short by 25 pcs.
['My shipment was short by', 25, '']
- qty: 25
- text1: ['My shipment was short by']
- text2: 

EDIT: Pyparsing supports two forms of alternatives for matching: MatchFirst, which stops on the first matched alternative (which is defined using the '|' operator), and Or, which evaluates all alternatives and selects the longest match (defined using '^' operator). So if you need a priority of the quantity expression, then you define it explicitly:

qty_pcs_expr = int_qty("qty") + White().suppress() + pieces.suppress()
qty_expr = Optional(LOT + OF).suppress() + int_qty("qty") + FollowedBy(White())

# compose expression for entire line
line_expr = (SkipTo(qty_pcs_expr)("text1") + qty_pcs_expr + restOfLine("text2") |
             SkipTo(qty_expr)("text1") + qty_expr + restOfLine("text2"))

Here are the new tests:

tests = """
    Send me 1000 widgets pronto!
    Deliver a Lot of 50 barrels of maple syrup by Monday, June 10.
    My shipment was short by 25 pcs.
    2. I expect 22 pcs delivered in the morning
    On May 15 please deliver 1000 PCS.
    """

Giving:

2. I expect 22 pcs delivered in the morning
['2. I expect ', 22, ' delivered in the morning']
- qty: 22
- text1: ['2. I expect ']
- text2:  delivered in the morning


On May 15 please deliver 1000 PCS.
['On May 15 please deliver ', 1000, '']
- qty: 1000
- text1: ['On May 15 please deliver ']
- text2: 

Answer 2

I don't know if you want to use re, but here's a regular expression which I think works. You can change the str value to test it. The match returns a tuple which has the three values [{text1}]{quantity}[{text2}]. The first and last items in the tuple will be empty if text1 and text2 are missing.

import re

str = "aSOETIHSIBSROG1PCS.ecsrGIR"

matchObj = re.search(r'([a-zA-Z]+|)(\dPCS?\.?|Lot of \d)([a-zA-Z]+|)',str).groups()
print matchObj.groups()

#Output
('aSOETIHSIBSROG', '1PCS.', 'ecsrGIR')