ERR_PROXY_CONNECTION_FAILED when using selenium-wire

Question

I tried to do the usual task to scrape data from a website.

Because I found that the data in the website is actually HTTP requests, and I can see the json in the response, I want to scrape the json files directly.

Then I found this selenium-wire which said "Extends Selenium to give you the ability to inspect requests made by the browser."

It works as expected in the first page. I got the json file. But whenever I click the link in the webpage using the webdriver, the connection is broken, saying ERR_PROXY_CONNECTION_FAILED.

I tried to switch back to selenium. It works again (without the json download).

So, are there any potential problems I can check with? and

Any other way to get the json? (Request seems not working because the websites need log in)

Answer 1

I found out that I'm getting this error if script finished execution. I put time.sleep(1000) at the end of the script. While a script is still running, link clicks and opening new pages work normally.