Encoding problem in Python and how to solve it

date: Nov 09, 2017
- Python
language: English

I recently meet a bug that really annoys me when trying to make a web spider. The issue is that the received html data is returned value is incorrectly processed. One example is the non-ascii right-single quotation mark’’ '. The expected value should be something like this:

{"description": "The New Topographic photographers acknowledged people\u2019s interaction with...

However, I got this:

{"description": "The New Topographic photographers acknowledged people\u00e2\u0080\u0099s interaction with...

This is because Python parse string data into byte arrays, for example, People’ s will be converted to a bytestring of ['\u0070', '\u0065', '\u006f', '\u0070', \u006c', '\u0065', '\u00e2\u0080\u0099', '\u0073']. When people try to read the bytestring as string, Python will not consider \u00e2\u0080\u0099 as a single character but three seperate character.

To Solve this problem, we need to identify “UTF-8-like” Unicode sequences and process them into the correct Unicode character using regular expressions:

import re

output_string = re.sub(ur'[\xc2-\xf4][\x80-\xbf]+',lambda m: m.group(0).encode('latin1').decode('utf8'),input_string)

output_string = re.sub(r'[\xc2-\xf4][\x80-\xbf]+',lambda m: m.group(0).encode('latin1').decode('utf8'),input_string)


Creative Commons License

Unless specified, all the articles are licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

It uses a self-hosted Open Web Analytics to get insights. Click here to optout.

The site is powered by Hexo. The source code of this website is here, so that you can see the spaghetti code behind my blog.

Main | Blog | Comic | Podcasts (Preparing) | RSS | Sitemap
2008 - 2022 | Patrick WU Jinming