35

Later today or tomorrow, we will be moving the API behind our Content Delivery Network (CDN). In an effort to better align all our applications, we are moving this and, in the future, other public domains behind the CDN. Ultimately, this helps us simplify our overall configuration and gives us the option to leverage additional features that the CDN provides, such as DDoS protection.

We don’t anticipate this impacting the network or the average user experience, but we wanted to let you know so you are aware. You might notice that our IP address for the API has changed, but that should be the only thing you notice.

The only possible impact we anticipate is for someone who is running any scripts that are hitting the API a ton, but in those cases, there is a good chance they might have been hitting the API a bit more frequently than we wanted anyway.

What happens if I notice some weirdness?

If you notice that something with the API is off or not working correctly, you should get an error message with a RayID from Cloudflare on your end. Please leave an answer to this question with the RayID and a short description of what you were trying to do with the API so we can investigate further if any issues arise.

Update April 25th 2024:

The migration has been postponed until Tuesday, April 30th.

Update April 30th 2024:

The migration has been completed. Logs show traffic is steady, we will continue monitoring traffic.

Update May 3rd 2024:

After a few days of monitoring, the move behind the CDN has not affected API traffic in any meaningful way externally.

17
  • 3
    The entire network was unavailable to me for something like 30 seconds around 25 minutes ago. Not an enormous glitch in the grand scheme of things but more services than usual were unavailable (chat often stays up when the others go down).
    – tripleee
    Commented Apr 24 at 13:13
  • 7
    If a user can't access MSE or MSO, they're not going to be able to reply or access the site - any alternatives for bug reports should things go horribly wrong? Commented Apr 24 at 13:33
  • 3
    @AaronBertrand Thanks, it wasn't just me though; several users in SOCVR apparently experienced it too. (All of us are probably in Europe.)
    – tripleee
    Commented Apr 24 at 13:37
  • 8
    There does seem to be a spike in errors around that timeframe: p.us3.datadoghq.com/sb/… however the API cutover is / will be unrelated to any of the SENetwork sites.
    – Josh Zhang StaffMod
    Commented Apr 24 at 14:11
  • 6
    What exactly is "a ton"? I host several things that use the API at a rate of roughly 1-2 requests per minute (with a peak of 1 request/6-8 seconds on average, not factoring in rate limiting, and slightly more if you count the average across all of them), and cloudflare can barely handle me logging into chat three times in a short period of time for bot testing without throwing a captcha at me
    – Zoe
    Commented Apr 24 at 16:03
  • 4
    We are not implementing any new rate limiters as part of the migration. API currently has an app based rate limiter that we'll be moving into Cloudflare but that's not going to be part of this move. I'm striving to make this change as transparent as possible by keeping the variables very limited. As far as blocking, the only thing I can think of that would be "new" block wise would be if someone doesn't send a user agent with their API call. This is a standard rule Cloudflare / many other CDN providers have.
    – Josh Zhang StaffMod
    Commented Apr 24 at 17:46
  • 1
    @Josh I mean blocks such as this one, just for the official API. Commented Apr 24 at 18:02
  • 1
    @ShadowWizardLoveZelda I suspect that issue shouldn't happen for API since we 100% expect only "bots" to be hitting that. For the main site, it could be some form of DDoS or anti-bot behavior baked into Cloudflare. Generally speaking we won't outright block scraping of the site but it's not "supported" since the API and data dump is there for a reason.
    – Josh Zhang StaffMod
    Commented Apr 24 at 18:40
  • 3
    @JoshZhang Some scraping is still forced by the API not supporting authentication (specifically for chatbots, where scraping the login page is required to access chat). Are you sure this isn't going to kill chatbots or make them flaky?
    – Zoe
    Commented Apr 24 at 18:45
  • 5
    I definitely can't be 100% certain as I'm not fully aware of every interaction chatbots will have with the site and the API. I believe the login page sits behind the SENetwork app, not api.stackexchange.com, so that isn't changing. Calls to api.stackexchange.com will be pointing to Cloudflare first instead of hitting our datacenters first. That will change. Cloudflare will have some normal threat detection like I mentioned but we are not adding any additional rules during this migration.
    – Josh Zhang StaffMod
    Commented Apr 24 at 19:05
  • 2
    @Ginger it does not.
    – Josh Zhang StaffMod
    Commented Apr 25 at 13:49
  • 7
    @JoshZhang Will there be any new requirements on the User-Agent passed with SE API requests? I ask because we (Charcoal/SD) encountered a problem earlier this month with CloudFlare silently giving erroneous, but successful (200) responses to requests to determine if a URL, https://t.ly/AqjA, is a redirect. With the User-Agent SmokeDetector/git (+https://github.com/Charcoal-SE/SmokeDetector) the test URL returned a 200, which is wrong, but when we changed the User-Agent to Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:123.0) Gecko/20100101 Firefox/123.0 it correctly returned a 302.
    – Makyen
    Commented Apr 25 at 17:42
  • 3
    @Makyen I'm unfamiliar with this particular behavior. I'll hop into the SmokeDetector channel in a bit to discuss this further because I'd like to dig into this further.
    – Josh Zhang StaffMod
    Commented Apr 25 at 17:57
  • 1
    To address my first comment, 1-2 requests per minute (combined with standard traffic from userscripts and chat activity) is indeed too much. On 2024-05-29 at approximately 13:29 (all timestamps in UTC+2), a moderation tool I run was taken down for 6 minutes (operation restored at 13:35:39), and the comment archive was down for 31 minutes (operation restored at 14:00:40) because Cloudflare said no. This spiral seems to have started shortly after a reconnect spiral due to repeated chat socket disconnects in the time interval 12:59-13:20, as well as a few times prior to 12:59
    – Zoe
    Commented Jun 2 at 20:48
  • 1
    For Boson in particular, following the API block, it got stuck in a restart cycle that failed due to attempts to log into chat again. Worth noting that Boson's attempts were 30 seconds apart, because the faults I accounted for did not include Cloudflare being a pain. I'll be expanding my libraries with handlers to avoid repeated failed restart cycles, but this happening in the first place makes it significantly harder for me to run several bots and other API programs on my home IP, especially if Cloudflare is this aggressive
    – Zoe
    Commented Jun 2 at 20:50

1 Answer 1

12

I don't think this is sensible. CDNs are geographically-distributed caches: they make a lot of sense for things like the new i.sstatic.net image host, but a CDN-proxy makes no sense for an API like Stack Exchange's.

I'd strongly advise against putting Cloudflare's anti-bot / DDoS protection on the API. Cloudflare uses a load of (secret!) heuristics to determine whether a user is "legit", including:

  • IP address shenanigans.
  • User-agent header sniffing.
  • Extra query parameters (and how the user-agent handles those).
  • Extra cookies (and how the user-agent handles those).
  • Analysis of browsing behaviour (does it look statistically-typical?).
  • CPU-intensive JavaScript wall.
    • hCAPTCHA wall, if you don't pass the JavaScript wall convincingly / quickly enough.

I regularly fail these heuristics: there are some sites I just can't use any more because of Cloudflare. I expect your average API consumer (a semi-supervised computer program) to fail these heuristics much more often. A single failure means the client expecting specially-formatted JSON will receive a chunk of unrelated HTML, and will probably just fall over. Writing well-behaving code that can handle a Cloudflare-"protected" site is much harder than it needs to be. (Abuse is, however, easy: just run a headless copy of Chrome. The undetected-chromedriver package on PyPi (warning: flashing images) purports to do that.)

If you want DDoS protection for the API, you're going to have to roll your own (e.g. nginx + fail2ban), or find a supplier who'll let you customise the HTTP responses. If there was a plan to move the API to on-demand compute (e.g. Azure Functions) and the DDoS protection was to cap your costs… sorry, but that plan was doomed from the start. (For several other reasons, too: the cloud might make sense for Stack Overflow for Teams, depending on your demand-as-a-function-of-time, but it makes no sense for the Network.)

If you do end up putting the API behind Cloudflare's DDoS protection, I'd appreciate an example client on Stack Apps, demonstrating how to properly handle the Cloudflare errors. I know some workarounds, but I have nothing systematic enough to automate the process.

Using Cloudflare as a reverse proxy might be okay, but anything that adds the /cdn-cgi/ endpoint is likely to cause problems.

8
  • That doesn't sound good. But how is it consistent with "You might notice that our IP address for the API has changed, but that should be the only thing you notice" ?
    – PM 2Ring
    Commented Apr 27 at 20:38
  • 2
    @PM2Ring Hypotheses: (a) Somebody somewhere was misinformed at some point. (a.1) The people working on this have never encountered a Cloudflare false positive, so aren't familiar with its failure modes. (b) There's a special way of configuring Cloudflare that doesn't have these issues, which isn't documented anywhere on Cloudflare's site. (b.1) It is documented, and I can't read.
    – wizzwizz4
    Commented Apr 27 at 20:48
  • 1
    The decision has already been made, at first I wasn't sure if it's good or bad, you did convince me it's bad and I used the only weapon I have which is downvoting the announcement. Not that it will help, even with 10000 downvotes, they would still do it, as they proved way too many times. Anyhow, thanks for the explanation and the nice way of putting it. Commented Apr 28 at 8:02
  • 5
    For the sake of transparency I want to say that we're not moving API behind Cloudflare because we want the DDoS protection. One of the major factors is visibility for all our traffic. Very recently there was an issue where someone was sending malicious traffic to chat and the team had a very difficult time detecting and stopping the traffic. Moving all ingress behind the CDN will resolve that. While Cloudflare is a CDN with good DDoS protection, their service offers us a variety of useful tools to enhance how we run the site and its applications.
    – Josh Zhang StaffMod
    Commented Apr 28 at 19:30
  • @JoshZhang That should be possible without Cloudflare, but I understand the utility of having everything in one dashboard. Does that mean you can disable all Cloudflare's other stuff on the API domain, keeping only the monitoring / dumb firewall? If so, I don't anticipate the issues mentioned in my answer.
    – wizzwizz4
    Commented Apr 28 at 19:40
  • 5
    There are other things that are very handy like their load balancer which we use to dynamically send traffic to CO when NY is unstable. But to answer your question, yes for the migration I will not be turning anything else on besides the basic vulnerability based rules like block traffic that doesn't send a user agent, block fake googlebot, etc. Those rules are part of this: developers.cloudflare.com/waf/managed-rules. Not all the rules will be turned on, just the ones that are on all the other sites.
    – Josh Zhang StaffMod
    Commented Apr 28 at 20:16
  • 2
    @JoshZhang The rules on the other sites are already too much for some use-cases, e.g. the main sites set the __cf_bm cookie and expect me to send it back to the servers. My concern is that none of this is documented in the API, and this is a breaking change that'll kill some of the API consumers.
    – wizzwizz4
    Commented Apr 28 at 20:27
  • 2
    @wizzwizz4 The __cf_bm cookie is part of their proprietary algo to help w/ bot detection, DDoS, etc. If none of the rules are turned on, the cookie is then just a byproduct of having the traffic go through Cloudflare and nothing more.
    – Josh Zhang StaffMod
    Commented Apr 28 at 20:31

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .