Cannot replicate successful Postman request with HttpClient

Question

I want to download a web page using HttpClient in .NET, but I receive a 403 Forbidden response, whereas using Postman with the same request parameters, I get a 200 OK. Here's my C# code:

var client = new HttpClient();
client.DefaultRequestHeaders.UserAgent.ParseAdd("PostmanRuntime/7.44.0");

var requestUri = "https://www.proshop.no/LEGO/LEGO-Botanicals-10348-Bonsaitre-Japansk-loenn/3340246";
var request = new HttpRequestMessage(HttpMethod.Get, requestUri);
var response = await client.SendAsync(request);
Console.WriteLine(response.StatusCode);

Here's the debug view:

And here's Postman's view:

I've tried adding more headers to HttpClient, like Accept, Accept-Language, and a few others, but that didn't help. The only thing that is definitely required in Postman's case is the User-Agent; otherwise, it returned 403 as well. I also tried a different HTTP version for HttpClient.

My main question is, what else does Postman do that HttpClient does not, which might be the reason for this behavior?

Answer 1

I tried your code out, and it turns out that the page is protected by Cloudflare.

Getting past Cloudflare's bot protection requires more realistic browser behavior, adding more browser-like headers is what got me past, see below:

client.DefaultRequestHeaders.Add("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36");
client.DefaultRequestHeaders.Add("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8");
client.DefaultRequestHeaders.Add("Accept-Language", "en-US,en;q=0.5");
client.DefaultRequestHeaders.Add("Sec-Fetch-Dest", "document");

This is not a specific headers combo, you'll see that other headers work as well or that you might have to add more, you just need enough to convince Cloudflare's bot protection that you aren't a bot.