Python UnicodeEncodeError, but I have encoded the parameters to UTF-8

Question

Here is my code:

def renren_get_sig(params):
    cat_params = ''.join([u'%s=%s'%(unicode(k), unicode(params[k])) for k in sorted(params)])
    sig = hashlib.md5(u"%s%s"%(unicode(cat_params), unicode(SEC_KEY))).hexdigest()
    return sig

The exception message is:

Exception Type: UnicodeEncodeError
Exception Value: 'ascii' codec can't encode characters in position 138-141: ordinal not in range(128)

The dic params value is as the following:

params ={
'access_token':u'195036|6.3cf38700f.2592000.1347375600-462350295',
 'action_link': u'http://wohenchun.xxx.com',
 'action_name': u'\u6d4b\u8bd5\u4e00\u4e0b',
 'api_key': u'8c0a2cded4f84bbba4328ccba22c3374',
 'caption': u'\u7eaf\u6d01\u6307\u6570\u6d4b\u8bd5',
 'description': u'\u4e16\u754c\u8fd9\u4e48\u4e71\uff0c\u88c5\u7eaf\u7ed9\u8c01\u770b\uff1f\u5230\u5e95\u4f60\u6709\u591a\u5355\u7eaf\uff0c\u8d85\u7ea7\u5185\u6db5\u7684\u4f60\uff0c\u6562\u4e0d\u6562\u6311\u6218\u8d85\u7ea7\u5185\u6db5\u7684\u9898\u76ee?\u4e0d\u7ba1\u4f60\u6d4b\u4e0d\u6d4b\uff0c\u53cd\u6b63\u6211\u662f\u6d4b\u4e86\uff01',
 'format': u'JSON',
 'image': u'http://hdn.xnimg.cn/photos/hdn21/20120809/1440/h0dd1376.jpg',
 'message': u'\u5c3c\u?!! \u3010\u4f60\u96be\u9053\u6bd4\u6211\u66f4\u7eaf\u6d01\u4e48,\u6765\u6d4b\u8bd5\u4e00\u4e0b\u5427!\u4f20\u9001\u95e8 >>  http://wohenchun.jiongceyan.com \u3011\r\n\t\t\t\t\t\t\t\t\t\t',
 'method': u'feed.publishFeed',
 'name': u'\u4eba\u4eba\u53f2\u4e0a\u6700\u706b\u7206\u6d4b\u8bd5\u4e4b\u5355\u7eaf\u6d4b\u8bd5',
 'url': u'http://wohenchun.xxx.com',
 'v': u'1.0'} 

All the key-value pairs in params are Unicode objects. Why do I still get such an exception?

Thank you!

Answer 1

Unicode is the problem. Hashing algorithms are designed to be used with bytes, not unicode code points. So you must choose encoding and encode your unicode strings to byte strings before applying hashing algorithm:

from hashlib import md5 

str_to_hash = unicode_str.encode('utf-8')
md5(str_to_hash).hexdigest()

There was an issue about this problem in Python tracker - investigate it for more information.