Split function add: \xef\xbb\xbf...\n to my list

Question

I want to open my file.txt and split all data from this file.

Here is my file.txt:

some_data1 some_data2 some_data3 some_data4 some_data5 

and here is my python code:

>>>file_txt = open("file.txt", 'r') >>>data = file_txt.read() >>>data_list = data.split(' ') >>>print data some_data1 some_data2 some_data3 some_data4 some_data5 >>>print data_list ['\xef\xbb\xbfsome_data1', 'some_data1', "some_data1", 'some_data1', 'some_data1\n'] 

As you can see here, when I print my data_list it adds to my list this: \xef\xbb\xbf and this: \n. What are these and how can I clean my list from them.

Thanks.

Answer 1

Your file contains UTF-8 BOM in the beginning.

To get rid of it, first decode your file contents to unicode.

fp = open("file.txt") data = fp.read().decode("utf-8-sig").encode("utf-8") 

But better don't encode it back to utf-8, but work with unicoded text. There is a good rule: decode all your input text data to unicode as soon as possible, and work only with unicode; and encode the output data to the required encoding as late as possible. This will save you from many headaches.

To read bigger files in a certain encoding, use io.open or codecs.open.

Also check this.

Use str.strip() or str.rstrip() to get rid of the newline character \n.