Selenium can not scrape Shopee e-commerce site using python

Question

I am not able to pull the price of products on Shopee (a e-commercial site).
I have taken a look at the problem solved by @dmitrybelyakov (link: Scraping AJAX e-commerce site using python) .

That solution helped me to get the 'name' of product and the 'historical_sold' but I can not get the price of the product. I can not find the price value in the Json string. Therefore, I tried to use selenium to pull data with xpath but it appeared to be failed.

The link of the ecommercial site: https://shopee.com.my/search?keyword=h370m

My code:

import time

from selenium import webdriver

import pandas as pd

path = r'C:\Users\\admin\\Desktop\\chromedriver_win32\\Chromedriver'

chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('headless')
chrome_options.add_argument('window-size=1200x600')

browserdriver = webdriver.Chrome(executable_path = path,options=chrome_options)
link='https://shopee.com.my/search?keyword=h370m'
browserdriver.get(link)
productprice='//*[@id="main"]/div/div[2]/div[2]/div/div/div/div[2]/div/div/div[2]/div[1]/div/a/div/div[2]/div[1]'
productprice_printout=browserdriver.find_element_by_xpath(productname).text
print(productprice_printout)

When I run that code, it showed the error notification like this:

selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//*[@id="main"]/div/div[2]/div[2]/div/div/div/div[2]/div/div/div[2]/div[1]/div/a/div/div[2]/div[1]"}

Please help me to get the price of product on Shopee!

Answer 1

You can use requests and the search API for the site

import requests

headers = {
    'User-Agent': 'Mozilla/5',
    'Referer': 'https://shopee.com.my/search?keyword=h370m'
}

url = 'https://shopee.com.my/api/v2/search_items/?by=relevancy&keyword=h370m&limit=50&newest=0&order=desc&page_type=search'  
r = requests.get(url, headers = headers).json()

for item in r['items']:
    print(item['name'], ' ', item['price'])

---

If you want roughly the same scale:

for item in r['items']:
    print(item['name'], ' ', 'RM' + str(item['price']/100000))