Why do I get a KeyError in this generator?

Question

I've got the following dictionary:

d = {
    'A': {
        'param': {
            '1': {
                'req': True,
            },
            '2': {
                'req': True,
            },
        },
    },
    'B': {
        'param': {
            '3': {
                'req': True,
            },
            '4': {
                'req': False,
            },
        },
    },
}

I want to have a generator which will give me for each first level keys, the required parameters.

req = {}
for key in d:
    req[key] = (p for p in d[key]['param'] if d[key]['param'][p].get('req', False))

So here, for each key in d, I get parameter p only if req is True.

However, when I try to use my generator, it raises a KeyError exception:

>>> req
{'A': <generator object <genexpr> at 0x27b8960>,
 'B': <generator object <genexpr> at 0x27b8910>}
>>> for elem in req['A']:
...     print elem
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-6-a96226f95cce> in <module>()
----> 1 for elem in req['A']:
      2     print elem
      3 

<ipython-input-4-1732088ccbdb> in <genexpr>((p,))
      1 for key in d:
----> 2         req[key] = (p for p in d[key]['param'] if d[key]['param'][p].get('req', False))
      3 

KeyError: '1'

Answer 1

The generator expressions you assign to req[key] binds on the key variable. But key changes from 'A' to 'B' in the loop. When you iterate over the first generator expression, it will evaluate key to 'B' in its if condition, even though key was 'A' when you created it.

The conventional way to bind to a variable's value and not its reference, is to wrap the expression in a lambda with a default value, and then call it immediately.

for key in d:
    req[key] = (lambda key=key: (p for p in d[key]['param'] if d[key]['param'][p].get('req', False)))()

Result:

1
2