Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Extracting dates from span using Python Selenium

I have this page:

Tripadvisor

For every review published there is a corresponding date in the title attribute,

check this :

<span class="ratingDate relativeDate" title="4 February 2017">Reviewed yesterday </span>

So, for every review published there is a date in the title attribute, my problem is that I am not able to fetch all the dates from the reviews.

I tried with this code:

def Dates():
datediv = driver.find_elements_by_css_selector('div > div.col2of2 > div > div.wrap > div.rating.reviewItemInline > span.ratingDate.relativeDate')
dateatt = datediv.get_Attribute("title")
for date in dateatt:
    print(date.text)

But still it does not work and I get the error of

AttributeError: 'list' object has no attribute 'get_Attribute'

Where am I going wrong?

Edit Ok so now I have scraped the Usernames, Date, Title and the entire review from every page, however, in just IDLE only. I want to put the scraped data from every page to say a dictionary and export it into json or maybe directly put it into an excel sheet.

The approach with the dictionary is quite confusing as I am literally not understand how would I update different keys independently with values.

content = {}

def mainfunction():
#Hotel Name
hname = driver.find_element_by_id('HEADING').text


#User Names
usernames = driver.find_elements_by_class_name('scrname')
for 

#Dates
datediv = driver.find_elements_by_css_selector('div > div.col2of2 > div > div.wrap > div.rating.reviewItemInline > span.ratingDate.relativeDate')


#Review Title
titlesdiv = driver.find_elements_by_class_name('isNew')
#for titles in titlesdiv:
#print(titles.find_element_by_class_name('noQuotes').text)


#Reviews 
linkdiv = driver.find_element_by_class_name('expandLink')
linkspan = linkdiv.find_element_by_class_name('ulBlueLinks')
linkspan.click()

try:
    WebDriverWait(driver,10).until(ec.presence_of_element_located((By.CLASS_NAME,"no_padding")))
    close1 = driver.find_element_by_css_selector('body > div> span > div.ui_close_x')
    close1.click()

except TimeoutException:
    print ("Loading took too much time!")


reviews = driver.find_elements_by_css_selector(' div > div.col2of2 > div > div.wrap > div > div > p')
for review in reviews:
    print(review.text)


#push the contents to the dictionary



#Move to next page
nextpage()


#To follow successive pages and scrape the content
def nextpage():
    nextpage = driver.find_element_by_css_selector('#REVIEWS >   div.deckTools.btm.test > div >   a.nav.next.rndBtn.ui_button.primary.taLnk').click()
    try:
        WebDriverWait(driver,10).until(ec.presence_of_element_located((By.CLASS_NAME,"pcb")))
        close2 = driver.find_element_by_class_name('ui_close_x')
    close2.click()
    except TimeoutException:
        print ("Loading took too much time!")

    mainfunction()
like image 868
raj shastri Avatar asked Apr 21 '26 01:04

raj shastri


1 Answers

datediv is list. You need to iterate over it

datediv = driver.find_elements_by_css_selector('div > div.col2of2 > div > div.wrap > div.rating.reviewItemInline > span.ratingDate.relativeDate')
for dateatt in datediv:
    print(dateatt.get_attribute("title"))
like image 72
Guy Avatar answered Apr 23 '26 15:04

Guy



Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!