How do I do multiple line list in ConfigArgParse?

Question

I have a config file with the following contents.

[Default]
file_list = ['dataset1/*.wav', 'dataset2/*.wav', 'dataset3/*.wav', 
'dataset4/*.wav']

which is a line break that most of the time works in Python. I parse it with this

import configargparse
p = configargparse.ArgParser()
p.add('-c', '--my-config', required=True, is_config_file=True, help='config file path')
p.add('--file_list', action="append", required=False, help='path to genome file')
args = p.parse_args()

by making the appropriate call

python file.py --my-config=config.ini

The problem is (not surprisingly):

file.py: error: unrecognized arguments: --'dataset4/*.wav'] true

I assume this happens because it expects a new arg in the config file. I also tried different line continuation methods.

How can I input multiple line lists using configargparse? If I can't, what is a minimal alternative solution to config file list inputs?

Answer 1

I came across the same problem. I decided to modify the default parser a bit and create my parser which supports multiple lines. I only added a few lines of code. It looks something like this:

import configargparse
from collections import OrderedDict
import re


class LongLineConfigFileParser(configargparse.ConfigFileParser):
    """Based on a simplified subset of INI and YAML formats. Here is the
    supported syntax:


        # this is a comment
        ; this is also a comment (.ini style)
        ---            # lines that start with --- are ignored (yaml style)
        -------------------
        [section]      # .ini-style section names are treated as comments

        # how to specify a key-value pair (all of these are equivalent):
        name value     # key is case sensitive: "Name" isn't "name"
        name = value   # (.ini style)  (white space is ignored, so name = value same as name=value)
        name: value    # (yaml style)
        --name value   # (argparse style)

        # how to set a flag arg (eg. arg which has action="store_true")
        --name
        name
        name = True    # "True" and "true" are the same


        # how to specify a list arg (eg. arg which has action="append")
        fruit = [apple, orange, lemon]
        indexes = [1, 12, 35 , 40]

        # how to break a long line into multiple lines:
        fruit = [apple, \
        orage, \
        lemon]


    """

    def parse(self, stream):
        """Parses the keys + values from a config file."""
        items = OrderedDict()
        pre_line = '' # Used to support multiple lines
        for i, line in enumerate(stream):
            line = pre_line + line
            line = line.strip()
            if not line or line[0] in ["#", ";", "["] or line.startswith("---"):
                continue
            # Used to support multiple lines
            if line.endswith('\\'):
                pre_line = line.replace('\\', '').strip()
                continue
            else:
                pre_line = ''
            white_space = "\\s*"
            key = r"(?P<key>[^:=;#\s]+?)"
            value = white_space+r"[:=\s]"+white_space+"(?P<value>.+?)"
            comment = white_space+"(?P<comment>\\s[;#].*)?"

            key_only_match = re.match("^" + key + comment + "$", line)
            if key_only_match:
                key = key_only_match.group("key")
                items[key] = "true"
                continue

            key_value_match = re.match("^"+key+value+comment+"$", line)
            if key_value_match:
                key = key_value_match.group("key")
                value = key_value_match.group("value")

                if value.startswith("[") and value.endswith("]"):
                    # handle special case of lists
                    value = [elem.strip() for elem in value[1:-1].split(",")]

                items[key] = value
                continue

            raise configargparse.ConfigFileParserException("Unexpected line {} in {}: {}".format(i,
                getattr(stream, 'name', 'stream'), line))
        return items

To use it you only need to add config_file_parser_class=LongLineConfigFileParser when calling the ArgParser:

parser = configargparse.ArgParser(default_config_files=['DefaultParams.txt'],
                                  config_file_parser_class=LongLineConfigFileParser)

Hope this helps