Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Python convert html ascii encoded text to utf8

I have a xml file, which I need to convert to utf8. Unfortunately the entities contain text like this:

/mytext,

I'm using the codec library to convert files to utf8, but html entities won't work with it.

Is there an easy way to get rid of the html encoding?

Thanks

like image 355
thesonix Avatar asked Jul 10 '26 01:07

thesonix


2 Answers

You can pass the text of the file through an unescape function before passing it to the XML parser.

Alternatively, if you're only parsing HTML, lxml's http parser does this for you:

>>> import lxml.html
>>> html = lxml.html.fromstring("<html><body><p>&#047;mytext&#044;</p></body></html>")
>>> lxml.html.tostring(html)
'<html><body><p>/mytext,</p></body></html>'
like image 64
jterrace Avatar answered Jul 11 '26 14:07

jterrace


Recently posted the below in response to a similar question:

import HTMLParser     # html.parser in Python 3
h = HTMLParser.HTMLParser()
h.unescape('&#047;mytext&#044;')

Technically this method is "internal" and undocumented, but it's been in the API quite a while and isn't marked with a leading underscore.

Found it here; other approaches are also mentioned, of which BeautifulSoup is probably the best if you don't mind its "heaviness."

like image 22
kindall Avatar answered Jul 11 '26 15:07

kindall



Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!