50

Today, SmokeDetector/metasmoke was unable to raise either automatic or manual flags due to Cloudflare asking for human verification on the SE API for a significant period of time.

The response from Cloudflare we saw, multiple times, when we attempted to use the SE API to raise flags was:

<!DOCTYPE html><html lang="en-US"><head><title>Just a moment...</title><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><meta http-equiv="X-UA-Compatible" content="IE=Edge"><meta name="robots" content="noindex,nofollow"><meta name="viewport" content="width=device-width,initial-scale=1"><style>*{box-sizing:border-box;margin:0;padding:0}html{line-height:1.15;-webkit-text-size-adjust:100%;color:#313131}button,html{font-family:system-ui,-apple-system,BlinkMacSystemFont,Segoe UI,Roboto,Helvetica Neue,Arial,Noto Sans,sans-serif,Apple Color Emoji,Segoe UI Emoji,Segoe UI Symbol,Noto Color Emoji}@media (prefers-color-scheme:dark){body{background-color:#222;color:#d9d9d9}body a{color:#fff}body a:hover{color:#ee730a;text-decoration:underline}body .lds-ring div{border-color:#999 transparent transparent}body .font-red{color:#b20f03}body .pow-button{background-color:#4693ff;color:#1d1d1d}body #challenge-success-text{background-image:url(data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHdpZHRoPSIzMiIgaGVpZ2h0PSIzMiIgZmlsbD0ibm9uZSIgdmlld0JveD0iMCAwIDI2IDI2Ij48cGF0aCBmaWxsPSIjZDlkOWQ5IiBkPSJNMTMgMGExMyAxMyAwIDEgMCAwIDI2IDEzIDEzIDAgMCAwIDAtMjZtMCAyNGExMSAxMSAwIDEgMSAwLTIyIDExIDExIDAgMCAxIDAgMjIiLz48cGF0aCBmaWxsPSIjZDlkOWQ5IiBkPSJtMTAuOTU1IDE2LjA1NS0zLjk1LTQuMTI1LTEuNDQ1IDEuMzg1IDUuMzcgNS42MSA5LjQ5NS05LjYtMS40Mi0xLjQwNXoiLz48L3N2Zz4=)}body #challenge-error-text{background-image:url(data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHdpZHRoPSIzMiIgaGVpZ2h0PSIzMiIgZmlsbD0ibm9uZSI+PHBhdGggZmlsbD0iI0IyMEYwMyIgZD0iTTE2IDNhMTMgMTMgMCAxIDAgMTMgMTNBMTMuMDE1IDEzLjAxNSAwIDAgMCAxNiAzbTAgMjRhMTEgMTEgMCAxIDEgMTEtMTEgMTEuMDEgMTEuMDEgMCAwIDEtMTEgMTEiLz48cGF0aCBmaWxsPSIjQjIwRjAzIiBkPSJNMTcuMDM4IDE4LjYxNUgxNC44N0wxNC41NjMgOS41aDIuNzgzem0tMS4wODQgMS40MjdxLjY2IDAgMS4wNTcuMzg4LjQwNy4zODkuNDA3Ljk5NCAwIC41OTYtLjQwNy45ODQtLjM5Ny4zOS0xLjA1Ny4zODktLjY1IDAtMS4wNTYtLjM4OS0uMzk4LS4zODktLjM5OC0uOTg0IDAtLjU5Ny4zOTgtLjk4NS40MDYtLjM5NyAxLjA1Ni0uMzk3Ii8+PC9zdmc+)}}body{display:flex;flex-direction:column;min-height:100vh}body.no-js .loading-spinner{visibility:hidden}body.no-js .challenge-running{display:none}body.dark{background-color:#222;color:#d9d9d9}body.dark a{color:#fff}body.dark a:hover{color:#ee730a;text-decoration:underline}body.dark .lds-ring div{border-color:#999 transparent transparent}body.dark .font-red{color:#b20f03}body.dark .pow-button{background-color:#4693ff;color:#1d1d1d}body.dark #challenge-success-text{background-image:url(data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHdpZHRoPSIzMiIgaGVpZ2h0PSIzMiIgZmlsbD0ibm9uZSIgdmlld0JveD0iMCAwIDI2IDI2Ij48cGF0aCBmaWxsPSIjZDlkOWQ5IiBkPSJNMTMgMGExMyAxMyAwIDEgMCAwIDI2IDEzIDEzIDAgMCAwIDAtMjZtMCAyNGExMSAxMSAwIDEgMSAwLTIyIDExIDExIDAgMCAxIDAgMjIiLz48cGF0aCBmaWxsPSIjZDlkOWQ5IiBkPSJtMTAuOTU1IDE2LjA1NS0zLjk1LTQuMTI1LTEuNDQ1IDEuMzg1IDUuMzcgNS42MSA5LjQ5NS05LjYtMS40Mi0xLjQwNXoiLz48L3N2Zz4=)}body.dark #challenge-error-text{background-image:url(data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHdpZHRoPSIzMiIgaGVpZ2h0PSIzMiIgZmlsbD0ibm9uZSI+PHBhdGggZmlsbD0iI0IyMEYwMyIgZD0iTTE2IDNhMTMgMTMgMCAxIDAgMTMgMTNBMTMuMDE1IDEzLjAxNSAwIDAgMCAxNiAzbTAgMjRhMTEgMTEgMCAxIDEgMTEtMTEgMTEuMDEgMTEuMDEgMCAwIDEtMTEgMTEiLz48cGF0aCBmaWxsPSIjQjIwRjAzIiBkPSJNMTcuMDM4IDE4LjYxNUgxNC44N0wxNC41NjMgOS41aDIuNzgzem0tMS4wODQgMS40MjdxLjY2IDAgMS4wNTcuMzg4LjQwNy4zODkuNDA3Ljk5NCAwIC41OTYtLjQwNy45ODQtLjM5Ny4zOS0xLjA1Ny4zODktLjY1IDAtMS4wNTYtLjM4OS0uMzk4LS4zODktLjM5OC0uOTg0IDAtLjU5Ny4zOTgtLjk4NS40MDYtLjM5NyAxLjA1Ni0uMzk3Ii8+PC9zdmc+)}body.light{background-color:transparent;color:#313131}body.light a{color:#0051c3}body.light a:hover{color:#ee730a;text-decoration:underline}body.light .lds-ring div{border-color:#595959 transparent transparent}body.light .font-red{color:#fc574a}body.light .pow-button{background-color:#003681;border-color:#003681;color:#fff}body.light #challenge-success-text{background-image:url(data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHdpZHRoPSIzMiIgaGVpZ2h0PSIzMiIgZmlsbD0ibm9uZSIgdmlld0JveD0iMCAwIDI2IDI2Ij48cGF0aCBmaWxsPSIjMzEzMTMxIiBkPSJNMTMgMGExMyAxMyAwIDEgMCAwIDI2IDEzIDEzIDAgMCAwIDAtMjZtMCAyNGExMSAxMSAwIDEgMSAwLTIyIDExIDExIDAgMCAxIDAgMjIiLz48cGF0aCBmaWxsPSIjMzEzMTMxIiBkPSJtMTAuOTU1IDE2LjA1NS0zLjk1LTQuMTI1LTEuNDQ1IDEuMzg1IDUuMzcgNS42MSA5LjQ5NS05LjYtMS40Mi0xLjQwNXoiLz48L3N2Zz4=)}body.light #challenge-error-text{background-image:url(data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHdpZHRoPSIzMiIgaGVpZ2h0PSIzMiIgZmlsbD0ibm9uZSI+PHBhdGggZmlsbD0iI2ZjNTc0YSIgZD0iTTE2IDNhMTMgMTMgMCAxIDAgMTMgMTNBMTMuMDE1IDEzLjAxNSAwIDAgMCAxNiAzbTAgMjRhMTEgMTEgMCAxIDEgMTEtMTEgMTEuMDEgMTEuMDEgMCAwIDEtMTEgMTEiLz48cGF0aCBmaWxsPSIjZmM1NzRhIiBkPSJNMTcuMDM4IDE4LjYxNUgxNC44N0wxNC41NjMgOS41aDIuNzgzem0tMS4wODQgMS40MjdxLjY2IDAgMS4wNTcuMzg4LjQwNy4zODkuNDA3Ljk5NCAwIC41OTYtLjQwNy45ODQtLjM5Ny4zOS0xLjA1Ny4zODktLjY1IDAtMS4wNTYtLjM4OS0uMzk4LS4zODktLjM5OC0uOTg0IDAtLjU5Ny4zOTgtLjk4NS40MDYtLjM5NyAxLjA1Ni0uMzk3Ii8+PC9zdmc+)}a{background-color:transparent;color:#0051c3;text-decoration:none;transition:color .15s ease}a:hover{color:#ee730a;text-decoration:underline}.main-content{margin:8rem auto;max-width:60rem;width:100%}.heading-favicon{height:2rem;margin-right:.5rem;width:2rem}@media (width <= 720px){.main-content{margin-top:4rem}.heading-favicon{height:1.5rem;width:1.5rem}}.footer,.main-content{padding-left:1.5rem;padding-right:1.5rem}.main-wrapper{align-items:center;display:flex;flex:1;flex-direction:column}.font-red{color:#b20f03}.spacer{margin:2rem 0}.h1{font-size:2.5rem;font-weight:500;line-height:3.75rem}.h2{font-weight:500}.core-msg,.h2{font-size:1.5rem;line-height:2.25rem}.body-text,.core-msg{font-weight:400}.body-text{font-size:1rem;line-height:1.25rem}@media (width <= 720px){.h1{font-size:1.5rem;line-height:1.75rem}.h2{font-size:1.25rem}.core-msg,.h2{line-height:1.5rem}.core-msg{font-size:1rem}}#challenge-error-text{background-image:url(data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHdpZHRoPSIzMiIgaGVpZ2h0PSIzMiIgZmlsbD0ibm9uZSI+PHBhdGggZmlsbD0iI2ZjNTc0YSIgZD0iTTE2IDNhMTMgMTMgMCAxIDAgMTMgMTNBMTMuMDE1IDEzLjAxNSAwIDAgMCAxNiAzbTAgMjRhMTEgMTEgMCAxIDEgMTEtMTEgMTEuMDEgMTEuMDEgMCAwIDEtMTEgMTEiLz48cGF0aCBmaWxsPSIjZmM1NzRhIiBkPSJNMTcuMDM4IDE4LjYxNUgxNC44N0wxNC41NjMgOS41aDIuNzgzem0tMS4wODQgMS40MjdxLjY2IDAgMS4wNTcuMzg4LjQwNy4zODkuNDA3Ljk5NCAwIC41OTYtLjQwNy45ODQtLjM5Ny4zOS0xLjA1Ny4zODktLjY1IDAtMS4wNTYtLjM4OS0uMzk4LS4zODktLjM5OC0uOTg0IDAtLjU5Ny4zOTgtLjk4NS40MDYtLjM5NyAxLjA1Ni0uMzk3Ii8+PC9zdmc+);padding-left:34px}#challenge-error-text,#challenge-success-text{background-repeat:no-repeat;background-size:contain}#challenge-success-text{background-image:url(data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHdpZHRoPSIzMiIgaGVpZ2h0PSIzMiIgZmlsbD0ibm9uZSIgdmlld0JveD0iMCAwIDI2IDI2Ij48cGF0aCBmaWxsPSIjMzEzMTMxIiBkPSJNMTMgMGExMyAxMyAwIDEgMCAwIDI2IDEzIDEzIDAgMCAwIDAtMjZtMCAyNGExMSAxMSAwIDEgMSAwLTIyIDExIDExIDAgMCAxIDAgMjIiLz48cGF0aCBmaWxsPSIjMzEzMTMxIiBkPSJtMTAuOTU1IDE2LjA1NS0zLjk1LTQuMTI1LTEuNDQ1IDEuMzg1IDUuMzcgNS42MSA5LjQ5NS05LjYtMS40Mi0xLjQwNXoiLz48L3N2Zz4=);padding-left:42px}.text-center{text-align:center}.pow-button{background-color:#0051c3;border:.063rem solid #0051c3;border-radius:.313rem;color:#fff;font-size:.875rem;line-height:1.313rem;margin:2rem 0;padding:.375rem 1rem;transition-duration:.2s;transition-property:background-color,border-color,color;transition-timing-function:ease}.pow-button:hover{background-color:#003681;border-color:#003681;color:#fff;cursor:pointer}.footer{font-size:.75rem;line-height:1.125rem;margin:0 auto;max-width:60rem;width:100%}.footer-inner{border-top:1px solid #d9d9d9;padding-bottom:1rem;padding-top:1rem}.clearfix:after{clear:both;content:"";display:table}.clearfix .column{float:left;padding-right:1.5rem;width:50%}.diagnostic-wrapper{margin-bottom:.5rem}.footer .ray-id{text-align:center}.footer .ray-id code{font-family:monaco,courier,monospace}.core-msg,.zone-name-title{overflow-wrap:break-word}@media (width <= 720px){.diagnostic-wrapper{display:flex;flex-wrap:wrap;justify-content:center}.clearfix:after{clear:none;content:none;display:initial;text-align:center}.column{padding-bottom:2rem}.clearfix .column{float:none;padding:0;width:auto;word-break:keep-all}.zone-name-title{margin-bottom:1rem}}.loading-spinner{height:76.391px}.lds-ring{display:inline-block;position:relative}.lds-ring,.lds-ring div{height:1.875rem;width:1.875rem}.lds-ring div{animation:lds-ring 1.2s cubic-bezier(.5,0,.5,1) infinite;border:.3rem solid transparent;border-radius:50%;border-top-color:#313131;box-sizing:border-box;display:block;position:absolute}.lds-ring div:first-child{animation-delay:-.45s}.lds-ring div:nth-child(2){animation-delay:-.3s}.lds-ring div:nth-child(3){animation-delay:-.15s}@keyframes lds-ring{0%{transform:rotate(0)}to{transform:rotate(1turn)}}@media screen and (-ms-high-contrast:active),screen and (-ms-high-contrast:none){.main-wrapper,body{display:block}}.rtl .heading-favicon{margin-left:.5rem;margin-right:0}.rtl #challenge-success-text{background-position:100%;padding-left:0;padding-right:42px}.rtl #challenge-error-text{background-position:100%;padding-left:0;padding-right:34px}</style><meta http-equiv="refresh" content="375"></head><body class="no-js"><div class="main-wrapper" role="main"><div class="main-content"><noscript><div id="challenge-error-title"><div class="h2"><span id="challenge-error-text">Enable JavaScript and cookies to continue</span></div></div></noscript></div></div><script>(function() {window._cf_chl_opt = {cvId: '3',cZone: "api.stackexchange.com",cType: 'managed',cNounce: '52255',cRay: '',cHash: '',cUPMDTk: "\/2.2\/questions\/104360\/flags\/options?__cf_chl_tk=",cFPWv: 'g',cTTimeMs: '1000',cMTimeMs: '375000',cTplV: 5,cTplB: 'cf',cK: "visitor-time",fa: "\/2.2\/questions\/104360\/flags\/options?__cf_chl_f_tk=",md: "",mdrd: "",cRq: {ru: '',ra: '',rm: 'R0VU',d: '',t: '',cT: Math.floor(Date.now() / 1000),m: '',i1: '',i2: '',zh: '',uh: '',hh: '',}};var cpo = document.createElement('script');cpo.src = '/cdn-cgi/challenge-platform/h/g/orchestrate/chl_page/v1?ray=';window._cf_chl_opt.cOgUHash = location.hash === '' && location.href.indexOf('#') !== -1 ? '#' : location.hash;window._cf_chl_opt.cOgUQuery = location.search === '' && location.href.slice(0, location.href.length - window._cf_chl_opt.cOgUHash.length).indexOf('?') !== -1 ? '?' : location.search;if (window.history && window.history.replaceState) {var ogU = location.pathname + window._cf_chl_opt.cOgUQuery + window._cf_chl_opt.cOgUHash;history.replaceState(null, null, "\/2.2\/questions\/104360\/flags\/options?" + window._cf_chl_opt.cOgUHash);cpo.onload = function() {history.replaceState(null, null, ogU);}}document.getElementsByTagName('head')[0].appendChild(cpo);}());</script></body></html>

We were told that putting the SE API behind Cloudflare would not add any additional impact on the use of the SE API. I'm very disappointed that has not turned out to be accurate. It is ridiculous that an API is asking for human verification. The entire point of having the SE API is that it's used programmatically, often completely without human interaction. It is even more ridiculous that it is asking for that human verification by sending HTML, in response to an API request which is supposed to be guaranteed to return JSON, which is required to be displayed and executed in a browser for human consumption and interaction.

In the time that it took me to realize there was an issue, determine what the issue was, and write up this question to report the problem, Cloudflare stopped asking for human verification and is now allowing us to raise flags through the SE API. Overall, the blockage was for about 1/2 to 1 hour.

Now, on the bright side, the company has been working with Charcoal to have additional secure methods to ensure that the company is able to clearly and securely identify requests which are originating from Charcoal (SmokeDetector, metasmoke, and autoflagging). Doing so is going to take some work on our side, which isn't complete, yet. However, while that may alleviate this problem for Charcoal, it doesn't resolve it for other consumers of the SE API who are definitely not expecting to get an HTML response that needs to be viewed in a browser.

The absolute minimum that needs to happen is that this requirement to have a human available and for consumers of the API to have the ability to display and run code in a browser needs to be documented in the API documentation.

3
  • 25
    I don't think documenting this is enough, an API consumer is not a human and will likely fail any test like this by design. The API can certainly have various rate limits and anti-abuse mechanisms, but asking for human verification doesn't make any sense in this context as you mentioned. Commented May 12 at 22:12
  • 6
    Crickets: exhibit 1, exhibit 2, exhibit 3
    – rene
    Commented May 13 at 7:10
  • 10
    I have forwarded this internally. Hold tight as I get more information please :)
    – Bella_Blue StaffMod
    Commented May 13 at 14:11

1 Answer 1

11

Legitimate API requests should never get a human validation gate from Cloudflare.

Having said that, what happened is that Cloudflare detected abnormally high number of requests to the API from Tor which triggered some of their anti-botnet defenses.

Cloudflare security events

I have tuned the configuration further and it seems to have resolved the issues.

Cloudflare firewall telemetry

I would like to re-iterate, legitimate traffic to the API should never encounter any Cloudflare protections.

Traffic that triggers Cloudflare protections will get a 403 or 429 response but will also load the CAPTCHA or JavaScript challenge from Cloudflare, this is a byproduct of their DDoS / botnet protections. We do not expect any human interactions for the API so those pages should be treaded as just a general error / block.

12
  • 3
    Also, who decides what is considered "legitimate"? Commented May 13 at 17:20
  • 5
    I would lean on the TOS to classify what is legitimate or not. This instance was due to Cloudflare being overzealous and behaving in a way we did not want. We don't have a way to make this never happen again without completely turning off the protections Cloudflare offers us but reports are escalated and we make sure to address them like I did with this one.
    – Josh Zhang StaffMod
    Commented May 13 at 17:35
  • 9
    I'm no security expert, but my understanding is that unusual requests triggering Cloudflare protection is...a very normal thing. So, the thing you are saying should never happen is caused by a normal thing happening - doesn't that suggest something is pretty fundamentally broken? Why would an API call ever result in an ask for human verification? Commented May 13 at 17:37
  • 2
    I used the qualifier should never because never is too absolute. As to the protection triggering, Cloudflare's protection settings are not on and off, they can be tuned to best suit the application that sits behind it. The human verification is a byproduct of the configuration we use for the actual site. We decided it's better to just present the human verification whenever we want to block someone instead of a hard block so that it can still be bypassed if it's really a person. For the API this doesn't make any sense but it still serves the purpose of blocking the request.
    – Josh Zhang StaffMod
    Commented May 13 at 17:40
  • 12
    @JoshZhang I don't understand how presenting the human verification is "better" for an API. The human verification returns HTML content. Things interacting with the API are expecting JSON content. If you can't define a message and return it, documenting it in the API, I would expect something like a 429 Too Many Requests or just a generic 400 Bad Request with an empty body. But the best would be a 429 or 400 with some kind of details in the JSON response. Commented May 13 at 18:07
  • 9
    I very much agree with Thomas; "so that it can still be bypassed if it's really a person" seems... without wanting to be too brash, absurd. It's an API. Humans interfacing with it directly is arguably a misuse. APIs should, categorically, always respond in terms that machines interfacing with that API can read and interpret, i.e. backoff parameters, HTML response codes, etc. There should never be an expectation that a person could possibly be around to interact with the page, and planning on that really sounds ridiculous to me.
    – zcoop98
    Commented May 13 at 18:36
  • 4
    The human bypass behavior is meant for the main site itself that bled over to API. Regardless, those pages should still be returning 403 or 429 depending on situation we do not expect a human be around for any API interactions.
    – Josh Zhang StaffMod
    Commented May 13 at 18:45
  • 1
    @Josh Gotcha, maybe I misunderstood. It might be helpful to document that more explicitly in your answer then; what I took away was that the human bypass response was expected, if a bit too overzealous before tuning.
    – zcoop98
    Commented May 13 at 19:04
  • 12
    Why did a heuristic triggered by Tor traffic block SmokeDetector? I'm pretty sure it is not using Tor, so either Cloudflare misidentified the traffic source or applied a very broad ban based on Tor traffic. Both versions seem suboptimal to me. Commented May 13 at 21:33
  • @JoshZhang Possible idea: As you mentioned "The human bypass behavior is meant for the main site", would the Cloudflare API Shield, preferably in log mode, work here? I don't know enough about that to say, but it seems to be closer to what you're seeking than treating the API as "normal" traffic, just with less sensitive rules. You'd probably need/want to talk to Cloudflare (and Charcoal, hopefully) to see if it'd be viable, but maybe…?
    – cocomac
    Commented May 14 at 0:07
  • 2
    We don't treat the traffic like normal traffic, everything has been tuned down as much as possible to account for the fact it's API traffic. API shield wasn't turned on due to the possibility of adding another layer of blocking behavior.
    – Josh Zhang StaffMod
    Commented May 14 at 11:29
  • @ThomasOwens: According to RFC 9110, content negotiation only applies to GET with response codes 200, 206, and 304. For all other response codes, the official standard says the server can give you whatever it feels like and your client just has to cope with it. This is sensible, because no other response code actually returns a "resource" in the sense of HTTP and REST - errors are not resources, so (from the perspective of the standard) there is nothing to negotiate.
    – Kevin
    Commented May 18 at 5:38

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .