9

Since around 8/14/2024, our services requests to https://api.stackexchange.com/2.2/search are getting blocked by Cloudflare.

Despite not having any changes in the traffic pattern or the actual code.

After some investigation, we determined that this is because the User-Agent header is not specified in our requests - adding it works around the issue and requests succeed.

I am wondering if others are experiencing this or if there have been new policies added on Cloudflare side?

Improve this question
9
  • 3
    Could you share how you are making these requests (Python, curl, a browser, etc.)?
    – cocomac
    Commented Aug 14 at 22:40
  • I have Github action that started to error "403 Forbidden" on API requests too. Not sure if it's the same issue.
    – SmallSoft
    Commented Aug 15 at 3:02
  • @MSDN.WhiteKnight See githubstatus.com/incidents/kz4khcgdsfdv if you haven't already.
    – cocomac
    Commented Aug 15 at 3:14
  • @cocomac The requests are made using HttpClient from an ASP.NET application deployed to Azure
    – Simon Schlegel
    Commented Aug 16 at 0:31
  • 2
    Could you try again? There's a similar/possible dupe issue The API is down that included this symptom, and staff mentioned that it's been fixed.
    – Meta Andrew T.
    Commented Aug 16 at 5:01
  • Our service is still getting blocked. Perhaps, we can check if we have user agent as suggested in the answer below
    – Simon Schlegel
    Commented Aug 16 at 17:21
  • 3
    There's way too little information here in this question to be able to address or duplicate the issue. You need to give us enough information such that the issue can be duplicated. For example, trivially going to the SE API site and doing a /search for the JavaScript tag on SO doesn't fail. So, it's not that we aren't concerned or don't want this resolved. It's that you haven't given us the minimum amount of information needed.
    – Makyen
    Commented Aug 17 at 2:20
  • 3
    We need things like: the exact request you're making (excluding any token, obviously). How you are making the requests, which would include, or at least imply, most of the HTTP headers, as Cloudflare has demonstrated in the past with other requests that the headers matter (e.g. the User-Agent). The actual response you're getting from the SE API and/or Cloudflare (which may have tracking info that would help SE look into the issue). etc.
    – Makyen
    Commented Aug 17 at 2:23
  • 2
    We were able to reproduce the issue in postman by removing/adding User-Agent. Looks like the Cloudflare configuration was changed to block requests without that header.
    – Simon Schlegel
    Commented Aug 20 at 0:00

1 Answer 1

Reset to default
6

I had similar issue, requests to /posts/ API was erroring "403 Forbidden" from Github actions job. I'm using WebClient in .NET console application. I debugged it, and the same error was returned when running locally too. I tried adding user agent header, and it helped! They probably added some silly protection that considers every client with blank user agent as malicious. Not sure if it's coming from Cloudflare or Stackexchange servers.

So my fix is this:

webClient.Headers["User-Agent"] = "AppName";

Try something like this, maybe it will resolve your issue too. Previously it was working without setting UA, so it would be good to clarify whether this change is intended.

Improve this answer
3
  • 3
    Does it require a human-like user agent, or any user agent? I could understand it requiring a non-blank user agent, because well-behaved API consumers should generally be at least coarsely identifying themselves in case there’s some widespread misbehaviour or bug that needs to be blocked.
    – Jeremy
    Commented Aug 16 at 13:20
  • @Jeremy Yeah, non-blank UA is enough. This way it makes more sense. Maybe the most of API users already set it, and thats why they were not broken by this.
    – SmallSoft
    Commented Aug 16 at 18:49
  • 1
    While this might be useful, it's inappropriate to answer this question, as you're just guessing as to what the problem is that the question is reporting. The question doesn't have enough information to duplicate the issue, or to even know what's going on. they don't even say what error they are getting. It's even an assumption on our part that by "getting blocked by Cloudflare" they mean that they are getting an error.
    – Makyen
    Commented Aug 17 at 2:26

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .