**Python selenium error** :Message: stale element reference: element is not attached to the page document

Question

I am trying to fetch the tweets from a twitter url. But when I execute the code, the following error occurs some of the times :

StaleElementReferenceException: Message: stale element reference: element is not attached to the page document
  (Session info: chrome=84.0.4147.89)

The error points to this line :

---> 18         print(tweet.text)

This is the corresponding code :

import time
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import csv

browser = webdriver.Chrome('/Users/Suraj/Desktop/twitter/chromedriver')
hashtag = 'Candles4SSR'
url = 'https://twitter.com/hashtag/'+hashtag+'?src=hashtag_click'
browser.get(url)
time.sleep(1)
body = browser.find_element_by_tag_name('body')

tweet_contents = []
for _ in range(5):
    tweets = browser.find_elements_by_css_selector("[data-testid=\"tweet\"]")
    
    for tweet in tweets:
        print(tweet.text)
        tweet_contents += [tweet.text]
    body.send_keys(Keys.PAGE_DOWN)
    time.sleep(0.4)

Any help or suggestions would be really appreciated.

Answer 1

seems like it takes the tweets to load more than 0.4 seconds

i copy pasted the code with changing the sleep time from 0.4 to 2 seconds and it worked fine.

browser = webdriver.Chrome('/Users/Suraj/Desktop/twitter/chromedriver')
hashtag = 'Candles4SSR'
url = 'https://twitter.com/hashtag/' + hashtag + '?src=hashtag_click'
browser.get(url)
time.sleep(1)
body = browser.find_element_by_tag_name('body')

tweet_contents = []
for _ in range(5):
    tweets = browser.find_elements_by_css_selector("[data-testid=\"tweet\"]")

    for tweet in tweets:
        print(tweet.text)
        tweet_contents += [tweet.text]
    body.send_keys(Keys.PAGE_DOWN)
    time.sleep(2)

if you see it works on your env consider increasing the sleep time from 0.4 to some higher number.

another option is to use a retry mechanism.