Error: You don't have permission to access "url" on this server. in Beautiful soup

Question

from bs4 import BeautifulSoup
import requests

site = 'https://www.france24.com/en/'

# Get HTML Content
r = requests.get(site)
soup = BeautifulSoup(r.content, 'html.parser')

print(soup)

# Find the Headline HTML
media_list = soup.find_all('div', class_='t-content__section-pb')

print(media_list)

I am trying to get the HTML from France24 for a content aggregator I am working on. This code works for most of the websites I have tried, but for some reason for a few sites it returns

You don't have permission to access "http://www.france24.com/en/" on this server.<p>
Reference #18.87e82d17.1592371984.e5f81db

Answer 1

you can try with passing header info in request

site = 'https://www.france24.com/en/'
header = {
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36',
}
# Get HTML Content
r = requests.get(site, headers=header)

Answer 2

Try to ignore SSL certificates,

import urllib.request, urllib.parse, urllib.error
from bs4 import BeautifulSoup as bs
import ssl

# --- ignore ssl certificate ---
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE

url = 'https://www.france24.com/en/'
html = urllib.request.urlopen(url, context=ctx).read() 

soup = bs(html, 'html.parser') 
media_list = soup.find_all('div', class_='t-content__section-pb')

print(media_list)

Note: BTW this did not return any data for the requested class. So I used soup.find(attrs={'class':'t-content__section-pb'})

import urllib.request, urllib.parse, urllib.error
from bs4 import BeautifulSoup as bs
import ssl

# --- ignore ssl certificate ---
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE

url = 'https://www.france24.com/en/'
html = urllib.request.urlopen(url, context=ctx).read()

soup = bs(html, 'html.parser')
media_list = soup.find(attrs={"t-content__section-pb"})

print(media_list)

Sample output:

<section class="t-content__section-pb">
<div class="o-banana-split">
<div class="o-layout-list o-banana-split__main-articles">
<div class="o-layout-list__item o-layout-list__item--main-item">
<div class="m-item-list-article m-item-list-article--main-article">
<a href="/en/20200617-beijing-closes-schools-again-as-covid-19-cases-mount">
<div class="article__figure-wrapper">
<figure class="m-figure m-figure--16x9">
...

Hope this works for you...happy scraping :)