Encoding problem in Python and how to solve it

date: Nov 09, 2017

I recently meet a bug that really annoys me when trying to make a web spider. The issue is that the received html data is returned value is incorrectly processed. One example is the non-ascii right-single quotation mark’’ '. The expected value should be something like this:

{"description": "The New Topographic photographers acknowledged people\u2019s interaction with...

However, I got this:

{"description": "The New Topographic photographers acknowledged people\u00e2\u0080\u0099s interaction with...

This is because Python parse string data into byte arrays, for example, People’ s will be converted to a bytestring of ['\u0070', '\u0065', '\u006f', '\u0070', \u006c', '\u0065', '\u00e2\u0080\u0099', '\u0073']. When people try to read the bytestring as string, Python will not consider \u00e2\u0080\u0099 as a single character but three seperate character.

To Solve this problem, we need to identify “UTF-8-like” Unicode sequences and process them into the correct Unicode character using regular expressions:

import re

output_string = re.sub(ur'[\xc2-\xf4][\x80-\xbf]+',lambda m: m.group(0).encode('latin1').decode('utf8'),input_string)

output_string = re.sub(r'[\xc2-\xf4][\x80-\xbf]+',lambda m: m.group(0).encode('latin1').decode('utf8'),input_string)


Creative Commons License
Unless specified, all the articles are licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

Free Software Fundation Member

The site is proudly powered by Hexo hosted on GitHub Pages.

The site is powered by the glorious CSS from bettermotherf**kingwebsite.com with some tweaks. Censored the f-word for good.

Main | Access | CDN | RSS | Sitemap
2008 - 2020 | Patrick WU Jinming