Why Curl Request in PHP sometimes work and others do not?

Question

On a ubuntu server I'm running the following PHP code and sometimes it works and sometimes it does not. The parts of the code that are commented on have already tested all possible variations and were not successful.

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'https://www.americanas.com.br/');
//curl_setopt($ch, CURLOPT_CAINFO, '/etc/ssl/certs/ca-certificates.crt');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_VERBOSE, true);
curl_setopt($ch, CURLOPT_TIMEOUT, 5);
//curl_setopt($ch, CURLOPT_DNS_CACHE_TIMEOUT, 0);
//curl_setopt($ch, CURLOPT_FRESH_CONNECT, true);
curl_setopt($ch, CURLOPT_STDERR, fopen(dirname(__FILE__).'/errorlog.txt', 'w'));
//curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
//curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
//curl_setopt($ch, CURLOPT_SSLVERSION, 4);
$result = curl_exec($ch);


curl_exec($ch);
print_r(curl_getinfo($ch));
print_r(curl_error($ch));

Below is a snapshot of it that I was able to do with the return of two requests, one that worked and one that did not. I noticed that each request is directed to a different IP, and the server's certificate also changes.

On the same server, executing a corresponding request via the command line always works:

curl --verbose https://www.americanas.com.br

Can someone explain why?

Answer 1

After many tests I come to some conclusions.

Every structure of the site, it seems, is behind a cluster of servers, which explains the large amount of IPs when resolving his DNS in various DNS services.

This my particular problem was caused by the DNS server used by my hosting. It was returning an IP to the site that was not working with my cURL requests via PHP.

I was able to work around this problem by changing the DNS server of the requests:

curl_setopt($ch, CURLOPT_DNS_SERVERS, '1.1.1.1,8.8.8.8');

Manually selecting an IP also worked.

curl_setopt($ch, CURLOPT_RESOLVE, ['www.americanas.com.br:443:23.36.73.68']);

I learned many new things with this problem, I hope it can be useful for someone else.

Answer 2

It's not related to the server that executes curls requests as much it's related to the domain itself and dns of the domain like cloudflare also if website have check on http and https requests so it may block requests partially so you make request and it gives response while other one blocked