Searching a list of words from a large file in python

Question

I am new python. I have a list of words and a very large file. I would like to delete the lines in the file that contain a word from the list of words.

The list of words is given as sorted and can be fed during initialization time. I am trying to find the best approach to solve this problem. I'm doing a linear search right now and it is taking too much time.

Any suggestions?

Answer 1

you can use intersection from set theory to check whether the list of words and words from a line have anything in common.

list_of_words=[]
sett=set(list_of_words)
with open(inputfile) as f1,open(outputfile,'w') as f2:
    for line in f1:
        if len(set(line.split()).intersection(sett))>=1:
            pass
        else:
            f2.write(line)

Answer 2

If the source file contains only words separated by whitespace, you can use sets:

words = set(your_words_list)
for line in infile:
    if words.isdisjoint(line.split()):
        outfile.write(line)

Note that this doesn't handle punctuation, e.g. given words = ['foo', 'bar'] a line like foo, bar,stuff won't be removed. To handle this, you need regular expressions:

rr = r'\b(%s)\b' % '|'.join(your_words_list)
for line in infile:
    if not re.search(rr, line):
        outfile.write(line)