Cant scrape google search results with beautifulsoup

Question

I want to scrape google search results , but whenever i try to do so, the program returns an empty list

from bs4 import BeautifulSoup
import requests

keyWord = input("Input Your KeyWord :")

url = f'https://www.google.com/search?q={keyWord}'
src = requests.get(url).text
soup = BeautifulSoup(src, 'lxml')

container = soup.findAll('div', class_='g')

print(container)

Answer 1

Complementing Andrej Kesely's answer if you getting empty results you always can climb one div up or down to test and go from there.

Code (say you want to scrape title, summary and link):

from bs4 import BeautifulSoup
import requests
import json

headers = {
    'User-agent':
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.19582"
}

html = requests.get('https://www.google.com/search?q=ice cream',
                    headers=headers).text

soup = BeautifulSoup(html, 'lxml')

summary = []

for container in soup.findAll('div', class_='tF2Cxc'):
  heading = container.find('h3', class_='LC20lb DKV0Md').text
  article_summary = container.find('span', class_='aCOpRe').text
  link = container.find('a')['href']

  summary.append({
      'Heading': heading,
      'Article Summary': article_summary,
      'Link': link,
  })

print(json.dumps(summary, indent=2, ensure_ascii=False))

Portion of output:

[
  {
    "Heading": "Ice cream - Wikipedia",
    "Article Summary": "Ice cream (derived from earlier iced cream or cream ice) is a sweetened frozen food typically eaten as a snack or dessert. It may be made from dairy milk or cream and is flavoured with a sweetener, either sugar or an alternative, and any spice, such as cocoa or vanilla.",
    "Link": "https://en.wikipedia.org/wiki/Ice_cream"
  },
  {
    "Heading": "Jeni's Splendid Ice Creams",
    "Article Summary": "Jeni's Splendid Ice Cream, built from the ground up with superlative ingredients. Order online, visit a scoop shop, or find the closest place to buy Jeni's near you.",
    "Link": "https://jenis.com/"
  }
]

---

Alternatively, you can do this using Google Search Engine Results API from SerpApi. It's a paid API with a free trial.

import os
from serpapi import GoogleSearch

params = {
    "engine": "google",
    "q": "ice cream",
    "api_key": os.getenv("API_KEY"),
}

search = GoogleSearch(params)
results = search.get_dict()

for result in results["organic_results"]:
  print(f"Title: {result['title']}\nSummary: {result['snippet']}\nLink: {result['link']}\n")

Portion of the output:

Title: Ice cream - Wikipedia
Summary: Ice cream (derived from earlier iced cream or cream ice) is a sweetened frozen food typically eaten as a snack or dessert. It may be made from dairy milk or cream and is flavoured with a sweetener, either sugar or an alternative, and any spice, such as cocoa or vanilla.
Link: https://en.wikipedia.org/wiki/Ice_cream

Title: 6 Ice Cream Shops to Try in Salem, Massachusetts ...
Summary: 6 Ice Cream Shops to Try in Salem, Massachusetts · Maria's Sweet Somethings, 26 Front Street · Kakawa Chocolate House, 173 Essex Street · Melt ...
Link: https://www.salem.org/icecream/

Title: Melt Ice Cream - Salem
Summary: Homemade ice cream made on-site in Salem, MA. Bold innovative flavors, exceptional customer service, local ingredients.
Link: https://meltsalem.com/

Disclaimer, I work for SerpApi.