cloudscraper.exceptions.CloudflareChallengeError: Detected a Cloudflare version 2 challenge. Error when I used cloudscraper module with python

Question

so I'm trying to bypass the cloudflare protection of a website to scrape some items from them but the Cloudscraper python module is not working.

Whenever I run it, I receive this error:

cloudscraper.exceptions.CloudflareChallengeError: Detected a Cloudflare version 2 challenge, This feature is not available in the opensource (free) version.

Here is a simplified code I'm using:

import cloudscraper
from bs4 import BeautifulSoup as soup


url = "http://adventurequest.life/"
scraper = cloudscraper.create_scraper()
html = scraper.get(url).text
page_soup = soup(html, "html.parser")
print(page_soup)

Do you guys have any idea how to fix this?

Answer 1

The cloudscraper library do not provide the bypass for cloudfare version 2 captcha in the free version. So in order to scrape such sites, one of the alternatives is to use a third party captcha solver.

Cloud scraper currently supports the following provider:

• 2captcha
• anticaptcha
• CapMonster Cloud
• deathbycaptcha
• 9kw
• return_response

You can subscribe to their respective APIs and use the given API key with cloud scraper like the example in their README

scraper = cloudscraper.create_scraper(
  interpreter='nodejs',
  captcha={
    'provider': '2captcha',
    'api_key': 'your_2captcha_api_key'
  }
)

But in case you are still facing issues, you can try to continue with other Anti Bot Bypass providers. For example you can try using third party proxies with requests using

import requests
url = "https://the.url/to/scrape" 
proxy = "http://subscribed.proxy/" 
proxies = {"http": proxy, "https": proxy} 
response = requests.get(url, proxies=proxies, verify=False)
print(response.text)