Django & Suds: UnicodeEncodeError When Using QuerySets

Question

As any Python developer I messed around with Python's Unicode issue for years now. But now I got a situation that drives me nuts and I'm not able to solve it by myself. It already took 1 day now, including recherches ..

My setup is a little Django application that connects to a remote system via SOAP (using Suds), pulling some data and looking for it in Django's database:

from myapp.models import Customer
client = suds.client.Client(...)
customer = client.service.getCustomerByEmail('[email protected]')

type(customer.email): <class 'suds.sax.text.Text'>

customer_exists = Customer.objects.filter(email=customer.email)

Now the customer's email address has a German Umlaut ü, which lets Django raise an Exception as follows:

Traceback (most recent call last):
  File "run_anatomy_client.py", line 19, in <module>
    print client.main()
  File "/Users/user/Documents/workspace/Wawi/application/myapp/client.py", line 282, in main
    if not Customer.objects.filter(email=customer.email.encode('latin1')):
  File "/Users/user/Documents/workspace/Wawi/application/myapp/client.py", line 76, in sync_customer
    if not customer_exists:
  File "/Users/user/Documents/workspace/Wawi/pyenv/lib/python2.7/site-packages/django/db/models/query.py", line 113, in __nonzero__
    iter(self).next()
  File "/Users/user/Documents/workspace/Wawi/pyenv/lib/python2.7/site-packages/django/db/models/query.py", line 107, in _result_iter
    self._fill_cache()
  File "/Users/user/Documents/workspace/Wawi/pyenv/lib/python2.7/site-packages/django/db/models/query.py", line 772, in _fill_cache
    self._result_cache.append(self._iter.next())
  File "/Users/user/Documents/workspace/Wawi/pyenv/lib/python2.7/site-packages/django/db/models/query.py", line 273, in iterator
    for row in compiler.results_iter():
  File "/Users/user/Documents/workspace/Wawi/pyenv/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 680, in results_iter
    for rows in self.execute_sql(MULTI):
  File "/Users/user/Documents/workspace/Wawi/pyenv/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 735, in execute_sql
    cursor.execute(sql, params)
  File "/Users/user/Documents/workspace/Wawi/pyenv/lib/python2.7/site-packages/django/db/backends/util.py", line 43, in execute
    logger.debug('(%.3f) %s; args=%s' % (duration, sql, params),
UnicodeEncodeError: 'ascii' codec can't encode character u'\xfc' in position 28: ordinal not in range(128)

I already played with encode(), decode(), changed the coding of the source file as well as the database layout, which currently looks as follows:

mysql> show variables like '%character%';
+--------------------------+-----------------------------------------+
| Variable_name            | Value                                   |
+--------------------------+-----------------------------------------+
| character_set_client     | latin1                                  |
| character_set_connection | latin1                                  |
| character_set_database   | utf8                                    |
| character_set_filesystem | binary                                  |
| character_set_results    | latin1                                  |
| character_set_server     | latin1                                  |
| character_set_system     | utf8                                    |
| character_sets_dir       | /opt/local/share/mysql5/mysql/charsets/ |
+--------------------------+-----------------------------------------+
8 rows in set (0.00 sec)

The strange thing is - if I set a tracepoint and execute the exact same line in the Django shell, then it works just fine when using encode():

(Pdb) Customer.objects.filter(email=customer.email)
*** UnicodeEncodeError: 'ascii' codec can't encode character u'\xfc' in position 28:     ordinal not in range(128)
(Pdb) Customer.objects.filter(email=customer.email.encode('utf-8'))
[]

I'd be thankful for any hints..

Answer 1

suds.sax.text.Text inherits from unicode

class Text(unicode):
    """
    An XML text object used to represent text content.
    @ivar lang: The (optional) language flag.
    @type lang: bool
    @ivar escaped: The (optional) XML special character escaped flag.
    @type escaped: bool
    """

You can just encode to UTF-8 if you want to work with.

email = customer.email.encode("utf-8")
customer_exists = Customer.objects.filter(email=email)