Python 3 scraping with Bs4

Question

I am trying to scrape the HTML of this website: https://www.idealista.com/venta-viviendas/madrid-madrid/ with python 3 (using PyCharm). I am only interested in the prices of the houses so I narrow my search to certain spans like so:

import requests
from bs4 import BeautifulSoup


page = requests.get('https://www.idealista.com/venta-viviendas/madrid-madrid/')

soup = BeautifulSoup(page.text, 'html.parser')


prices=soup.findAll("span", {"class": "item-price h2-simulated"})

print(len(prices))
print(prices)

When I run it I get this: 0 []

Which means it isn't finding anything. Furthermore, if I print all of it: print(soup) I get very little html content for a page that big, so it's obviously not fetching everything.

Answer 1

The site is dynamic, thus, you will need to use a browser manipulation tool such as selenium:

from bs4 import BeautifulSoup as soup
from selenium import webdriver
import re, collections, itertools
d = webdriver.Chrome('/Users/jamespetullo/Downloads/chromedriver')
d.get('https://www.idealista.com/venta-viviendas/madrid-madrid/')
homes = soup(d.page_source, 'html.parser').find_all('div', {'class':'item-info-container'})
results = [i.find('div', {'class':re.compile('price\-row')}).text for i in homes]
price = collections.namedtuple('price', ['original', 'current', 'drop'])
new_prices = [list(filter(None, re.split('\s{2,}', i))) for i in results]
final_prices = [price(a, *[a for a, _ in itertools.zip_longest(b, [None, None])]) for a, *b in new_prices]

Output:

[price(original=' 369.000€', current='395.000 €', drop='7%'), price(original=' 1.250.000€ Garaje incluido ', current=None, drop=None), price(original=' 1.590.000€ Garaje incluido', current='1.650.000 €', drop='4%'), price(original=' 1.750.000€ Garaje incluido', current='1.875.000 €', drop='7%'), price(original=' 1.090.000€ Garaje incluido', current='1.195.000 €', drop='9%'), price(original=' 795.000€ ', current=None, drop=None), price(original=' 1.095.000€ Garaje incluido ', current=None, drop=None), price(original=' 355.000€ Garaje incluido ', current=None, drop=None), price(original=' 995.000€ Garaje incluido ', current=None, drop=None), price(original=' 1.130.000€ Garaje incluido', current='1.190.000 €', drop='5%'), price(original=' 850.000€ Garaje incluido ', current=None, drop=None), price(original=' 1.200.000€ Garaje incluido ', current=None, drop=None), price(original=' 990.000€ Garaje incluido ', current=None, drop=None), price(original=' 2.100.000€ ', current=None, drop=None), price(original=' 830.000€ Garaje incluido ', current=None, drop=None), price(original=' 2.390.000€ Garaje incluido ', current=None, drop=None), price(original=' 685.000€ ', current=None, drop=None), price(original=' 1.150.000€ Garaje incluido', current='1.200.000 €', drop='4%'), price(original=' 915.000€ ', current=None, drop=None), price(original=' 1.590.000€ Garaje incluido ', current=None, drop=None), price(original=' 625.000€ Garaje incluido', current='640.000 €', drop='2%'), price(original=' 735.000€', current='760.000 €', drop='3%'), price(original=' 890.000€ Garaje incluido', current='950.000 €', drop='6%'), price(original=' 925.000€', current='999.000 €', drop='7%'), price(original=' 975.000€', current='1.100.000 €', drop='11%'), price(original=' 850.000€ Garaje incluido', current='870.000 €', drop='2%'), price(original=' 1.200.000€ ', current=None, drop=None), price(original=' 1.500.000€ ', current=None, drop=None), price(original=' 1.200.000€ ', current=None, drop=None), price(original=' 1.359.000€ ', current=None, drop=None)]