What dependencies are required for Python to avoid UnicodeDecodeError?

Question

I am using the Plone collective's Python buildout. It builds various versions of Python including 2.6 and 2.7.

For some reason, when I try to do a release with 2.7 I get this error:

$ /Users/aclark/Developer/collective/python/python-2.7/bin/mkrelease -T -d local
…
adding 'plonetheme.grungeera-0.1.1/plonetheme.grungeera.egg-info/dependency_links.txt'
adding 'plonetheme.grungeera-0.1.1/plonetheme.grungeera.egg-info/entry_points.txt'
adding 'plonetheme.grungeera-0.1.1/plonetheme.grungeera.egg-info/namespace_packages.txt'
adding 'plonetheme.grungeera-0.1.1/plonetheme.grungeera.egg-info/not-zip-safe'
adding 'plonetheme.grungeera-0.1.1/plonetheme.grungeera.egg-info/PKG-INFO'
adding 'plonetheme.grungeera-0.1.1/plonetheme.grungeera.egg-info/requires.txt'
adding 'plonetheme.grungeera-0.1.1/plonetheme.grungeera.egg-info/SOURCES.txt'
adding 'plonetheme.grungeera-0.1.1/plonetheme.grungeera.egg-info/top_level.txt'
removing 'plonetheme.grungeera-0.1.1' (and everything under it)
running register
running check
Traceback (most recent call last):
  File "<string>", line 22, in <module>
  File "setup.py", line 36, in <module>
    """,
  File "/Users/aclark/Developer/collective/python/parts/opt/lib/python2.7/distutils/core.py", line 152, in setup
    dist.run_commands()
  File "/Users/aclark/Developer/collective/python/parts/opt/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/Users/aclark/Developer/collective/python/parts/opt/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/Users/aclark/Developer/collective/python/python-2.7/lib/python2.7/site-packages/distribute-0.6.15-py2.7.egg/setuptools/command/register.py", line 9, in run
    _register.run(self)
  File "/Users/aclark/Developer/collective/python/parts/opt/lib/python2.7/distutils/command/register.py", line 51, in run
    self.run_command(cmd_name)
  File "/Users/aclark/Developer/collective/python/parts/opt/lib/python2.7/distutils/cmd.py", line 326, in run_command
    self.distribution.run_command(command)
  File "/Users/aclark/Developer/collective/python/parts/opt/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/Users/aclark/Developer/collective/python/parts/opt/lib/python2.7/distutils/command/check.py", line 69, in run
    self.check_restructuredtext()
  File "/Users/aclark/Developer/collective/python/parts/opt/lib/python2.7/distutils/command/check.py", line 111, in check_restructuredtext
    for warning in self._check_rst_data(data):
  File "/Users/aclark/Developer/collective/python/parts/opt/lib/python2.7/distutils/command/check.py", line 138, in _check_rst_data
    parser.parse(data, document)
  File "/Users/aclark/Developer/collective/python/python-2.7/lib/python2.7/site-packages/docutils/parsers/rst/__init__.py", line 157, in parse
    self.statemachine.run(inputlines, document, inliner=self.inliner)
  File "/Users/aclark/Developer/collective/python/python-2.7/lib/python2.7/site-packages/docutils/parsers/rst/states.py", line 170, in run
    input_source=document['source'])
  File "/Users/aclark/Developer/collective/python/python-2.7/lib/python2.7/site-packages/docutils/statemachine.py", line 233, in run
    context, state, transitions)
  File "/Users/aclark/Developer/collective/python/python-2.7/lib/python2.7/site-packages/docutils/statemachine.py", line 454, in check_line
    return method(match, context, next_state)
  File "/Users/aclark/Developer/collective/python/python-2.7/lib/python2.7/site-packages/docutils/parsers/rst/states.py", line 2706, in underline
    self.section(title, source, style, lineno - 1, messages)
  File "/Users/aclark/Developer/collective/python/python-2.7/lib/python2.7/site-packages/docutils/parsers/rst/states.py", line 329, in section
    self.new_subsection(title, lineno, messages)
  File "/Users/aclark/Developer/collective/python/python-2.7/lib/python2.7/site-packages/docutils/parsers/rst/states.py", line 398, in new_subsection
    node=section_node, match_titles=1)
  File "/Users/aclark/Developer/collective/python/python-2.7/lib/python2.7/site-packages/docutils/parsers/rst/states.py", line 284, in nested_parse
    node=node, match_titles=match_titles)
  File "/Users/aclark/Developer/collective/python/python-2.7/lib/python2.7/site-packages/docutils/parsers/rst/states.py", line 195, in run
    results = StateMachineWS.run(self, input_lines, input_offset)
  File "/Users/aclark/Developer/collective/python/python-2.7/lib/python2.7/site-packages/docutils/statemachine.py", line 233, in run
    context, state, transitions)
  File "/Users/aclark/Developer/collective/python/python-2.7/lib/python2.7/site-packages/docutils/statemachine.py", line 454, in check_line
    return method(match, context, next_state)
  File "/Users/aclark/Developer/collective/python/python-2.7/lib/python2.7/site-packages/docutils/parsers/rst/states.py", line 2706, in underline
    self.section(title, source, style, lineno - 1, messages)
  File "/Users/aclark/Developer/collective/python/python-2.7/lib/python2.7/site-packages/docutils/parsers/rst/states.py", line 329, in section
    self.new_subsection(title, lineno, messages)
  File "/Users/aclark/Developer/collective/python/python-2.7/lib/python2.7/site-packages/docutils/parsers/rst/states.py", line 398, in new_subsection
    node=section_node, match_titles=1)
  File "/Users/aclark/Developer/collective/python/python-2.7/lib/python2.7/site-packages/docutils/parsers/rst/states.py", line 284, in nested_parse
    node=node, match_titles=match_titles)
  File "/Users/aclark/Developer/collective/python/python-2.7/lib/python2.7/site-packages/docutils/parsers/rst/states.py", line 195, in run
    results = StateMachineWS.run(self, input_lines, input_offset)
  File "/Users/aclark/Developer/collective/python/python-2.7/lib/python2.7/site-packages/docutils/statemachine.py", line 233, in run
    context, state, transitions)
  File "/Users/aclark/Developer/collective/python/python-2.7/lib/python2.7/site-packages/docutils/statemachine.py", line 454, in check_line
    return method(match, context, next_state)
  File "/Users/aclark/Developer/collective/python/python-2.7/lib/python2.7/site-packages/docutils/parsers/rst/states.py", line 2706, in underline
    self.section(title, source, style, lineno - 1, messages)
  File "/Users/aclark/Developer/collective/python/python-2.7/lib/python2.7/site-packages/docutils/parsers/rst/states.py", line 329, in section
    self.new_subsection(title, lineno, messages)
  File "/Users/aclark/Developer/collective/python/python-2.7/lib/python2.7/site-packages/docutils/parsers/rst/states.py", line 398, in new_subsection
    node=section_node, match_titles=1)
  File "/Users/aclark/Developer/collective/python/python-2.7/lib/python2.7/site-packages/docutils/parsers/rst/states.py", line 284, in nested_parse
    node=node, match_titles=match_titles)
  File "/Users/aclark/Developer/collective/python/python-2.7/lib/python2.7/site-packages/docutils/parsers/rst/states.py", line 195, in run
    results = StateMachineWS.run(self, input_lines, input_offset)
  File "/Users/aclark/Developer/collective/python/python-2.7/lib/python2.7/site-packages/docutils/statemachine.py", line 233, in run
    context, state, transitions)
  File "/Users/aclark/Developer/collective/python/python-2.7/lib/python2.7/site-packages/docutils/statemachine.py", line 454, in check_line
    return method(match, context, next_state)
  File "/Users/aclark/Developer/collective/python/python-2.7/lib/python2.7/site-packages/docutils/parsers/rst/states.py", line 2646, in blank
    self.parent += self.literal_block()
  File "/Users/aclark/Developer/collective/python/python-2.7/lib/python2.7/site-packages/docutils/parsers/rst/states.py", line 2740, in literal_block
    literal_block = nodes.literal_block(data, data)
  File "/Users/aclark/Developer/collective/python/python-2.7/lib/python2.7/site-packages/docutils/nodes.py", line 807, in __init__
    TextElement.__init__(self, rawsource, text, *children, **attributes)
  File "/Users/aclark/Developer/collective/python/python-2.7/lib/python2.7/site-packages/docutils/nodes.py", line 795, in __init__
    textnode = Text(text)
  File "/Users/aclark/Developer/collective/python/python-2.7/lib/python2.7/site-packages/docutils/nodes.py", line 331, in __new__
    return reprunicode.__new__(cls, data)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 32: ordinal not in range(128)
register failed

However if I use Python 2.6:

$ /Users/aclark/Developer/collective/python/python-2.6/bin/mkrelease -T -d local

No UnicodeDecodeError. Everything works as expected.

This normally would lead me to believe that something was wrong with the environment in which Python was compiled (Mac OS X), but in this case both 2.6 and 2.7 were compiled in the same environment yet one works and one does not.

What could possibly be going wrong here? What can I look/check for to figure this out?

Answer 1

Here is what I think differs: The version of docutils in both python installations.

There is still constant work being done on the docutils package and that project releases the current Subversion HEAD as a snapshot on a regular basis. The line numbers in your Python 2.7 traceback seem to match the latest snapshot (which according to the headers on the download was released the 1st of September 2010).

I am reasonably sure that the Python 2.6 site-packages directory differs from the one in your Python 2.7 installation.

Run:

diff -ur python-2.6/lib/python2.6/site-packages/docutils python-2.7/lib/python2.7/site-packages/docutils 

to confirm this. If I'm right, just copy over the version in the 2.6 tree to the 2.7 tree, and have Python 2.7 re-compile them on import, or use the compileall.py script:

python-2.7/bin/python python-2.7/lib/python2.7/compileall.py lib/python2.7/site-packages/docutils