Cloudflare challenge on API and WebSocket traffic used for AI Analysis

Question

Currently, there's a reproducible issue with Cloudflare blocking (via user/browser challenge as in this question) all API and WebSocket traffic from several sources. This is impacting the AI analysis tools that both I and another user have written for the Gen-AI Heuristics Group.

Other posts here about the issue seem to me to be related to authentication, but in this case very simple GET requests that don't require authentication are being blocked.

The problem is occurring with:

• Nushell on Linux. E.g.:

  http get -ef https://api.stackexchange.com/2.3/answers?order=desc&sort=activity&site=stackoverflow
  

• Python using websocket-client (the other user's system):

  import websocket
  socket = websocket.WebSocketApp("wss://qa.sockets.stackexchange.com")
  websocket.enableTrace(True)
  socket.run_forever()
  

This is reproducible from multiple systems and IP addresses, even on remote hosts which I've freshly created to reproduce the issue.

The API works from (oddly) Nushell for Windows as well as PowerShell (Invoke-WebRequest) and browsers.

In addition to trying a different system, I've tried:

• Changing the User-Agent request header
• With and without a key param on the API request
• Creating a new StackApp registration and trying a new key

Did some new fingerprinting go into effect with Cloudflare and the SE API's that is blocking us?

The AI-Heuristic team could use some urgent assistance on getting back to normal, as we are losing data (I won't go into detail publicly on what) and hindered in our ability to review AI-generated content.

This code was previously running and stopped working (for me, via API GET) at 2:48 PM UTC, and for the other user (via websocket-client) around 3:25 PM UTC.

---

Additional debugging details from the websocket response:

--- request header ---
GET / HTTP/1.1
Upgrade: websocket
Host: qa.sockets.stackexchange.com
Origin: https://qa.sockets.stackexchange.com
Sec-WebSocket-Key: gOhJJXt1D2r6VUcUPYHFeQ==
Sec-WebSocket-Version: 13
Connection: Upgrade


-----------------------
--- response header ---
HTTP/1.1 403 Forbidden
Date: Tue, 09 Jul 2024 22:15:04 GMT
Content-Type: text/html; charset=UTF-8
Content-Length: 15598
Connection: close
Accept-CH: Sec-CH-UA-Bitness, Sec-CH-UA-Arch, Sec-CH-UA-Full-Version, Sec-CH-UA-Mobile, Sec-CH-UA-Model, Sec-CH-UA-Platform-Version, Sec-CH-UA-Full-Version-List, Sec-CH-UA-Platform, Sec-CH-UA, UA-Bitness, UA-Arch, UA-Full-Version, UA-Mobile, UA-Model, UA-Platform-Version, UA-Platform, UA
Critical-CH: Sec-CH-UA-Bitness, Sec-CH-UA-Arch, Sec-CH-UA-Full-Version, Sec-CH-UA-Mobile, Sec-CH-UA-Model, Sec-CH-UA-Platform-Version, Sec-CH-UA-Full-Version-List, Sec-CH-UA-Platform, Sec-CH-UA, UA-Bitness, UA-Arch, UA-Full-Version, UA-Mobile, UA-Model, UA-Platform-Version, UA-Platform, UA
Cross-Origin-Embedder-Policy: require-corp
Cross-Origin-Opener-Policy: same-origin
Cross-Origin-Resource-Policy: same-origin
Origin-Agent-Cluster: ?1
Permissions-Policy: accelerometer=(),autoplay=(),browsing-topics=(),camera=(),clipboard-read=(),clipboard-write=(),geolocation=(),gyroscope=(),hid=(),interest-cohort=(),magnetometer=(),microphone=(),payment=(),publickey-credentials-get=(),screen-wake-lock=(),serial=(),sync-xhr=(),usb=()
Referrer-Policy: same-origin
X-Content-Options: nosniff
X-Frame-Options: SAMEORIGIN
cf-mitigated: challenge
cf-chl-out: LHO4XpX09AwG8Mlp4gzWLOc2jNeIohU6DY5X/rqfSE6KB56afHQPE10lfxQjJTH02e5+U1zRk1uG1Dmf4uC36gwMErb5wUYoKLiGdgDV3cR9vpyZNRYRG9MLVP04a480oICZGs1juv4WJbChUF4mhw==$U9e7rmVNAXg4fcYa2H5IBw==
Cache-Control: private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Expires: Thu, 01 Jan 1970 00:00:01 GMT
Set-Cookie: __cf_bm=Y3xQzwwFzwN9JTBowBao6p0hFfXy5TVXBazDqQQ_oFE-1720563304-1.0.1.1-WU76lEBQTdjMlxsW_ArVirEbsq6XkEQ8N3QQyuauLjAf1HeLHRxZFUisSYMVPVz.GgEbuLNc72rMKmHX7pwQjQ; path=/; expires=Tue, 09-Jul-24 22:45:04 GMT; domain=.qa.sockets.stackexchange.com; HttpOnly; Secure
Server: cloudflare
CF-RAY: 8a0babac2d18bf7c-ATL
-----------------------
Handshake status 403 Forbidden -+-+- {'date': 'Tue, 09 Jul 2024 22:15:04 GMT', 'content-type': 'text/html; charset=UTF-8', 'content-length': '15598', 'connection': 'close', 'accept-ch': 'Sec-CH-UA-Bitness, Sec-CH-UA-Arch, Sec-CH-UA-Full-Version, Sec-CH-UA-Mobile, Sec-CH-UA-Model, Sec-CH-UA-Platform-Version, Sec-CH-UA-Full-Version-List, Sec-CH-UA-Platform, Sec-CH-UA, UA-Bitness, UA-Arch, UA-Full-Version, UA-Mobile, UA-Model, UA-Platform-Version, UA-Platform, UA', 'critical-ch': 'Sec-CH-UA-Bitness, Sec-CH-UA-Arch, Sec-CH-UA-Full-Version, Sec-CH-UA-Mobile, Sec-CH-UA-Model, Sec-CH-UA-Platform-Version, Sec-CH-UA-Full-Version-List, Sec-CH-UA-Platform, Sec-CH-UA, UA-Bitness, UA-Arch, UA-Full-Version, UA-Mobile, UA-Model, UA-Platform-Version, UA-Platform, UA', 'cross-origin-embedder-policy': 'require-corp', 'cross-origin-opener-policy': 'same-origin', 'cross-origin-resource-policy': 'same-origin', 'origin-agent-cluster': '?1', 'permissions-policy': 'accelerometer=(),autoplay=(),browsing-topics=(),camera=(),clipboard-read=(),clipboard-write=(),geolocation=(),gyroscope=(),hid=(),interest-cohort=(),magnetometer=(),microphone=(),payment=(),publickey-credentials-get=(),screen-wake-lock=(),serial=(),sync-xhr=(),usb=()', 'referrer-policy': 'same-origin', 'x-content-options': 'nosniff', 'x-frame-options': 'SAMEORIGIN', 'cf-mitigated': 'challenge', 'cf-chl-out': 'LHO4XpX09AwG8Mlp4gzWLOc2jNeIohU6DY5X/rqfSE6KB56afHQPE10lfxQjJTH02e5+U1zRk1uG1Dmf4uC36gwMErb5wUYoKLiGdgDV3cR9vpyZNRYRG9MLVP04a480oICZGs1juv4WJbChUF4mhw==$U9e7rmVNAXg4fcYa2H5IBw==', 'cache-control': 'private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0', 'expires': 'Thu, 01 Jan 1970 00:00:01 GMT', 'set-cookie': '__cf_bm=Y3xQzwwFzwN9JTBowBao6p0hFfXy5TVXBazDqQQ_oFE-1720563304-1.0.1.1-WU76lEBQTdjMlxsW_ArVirEbsq6XkEQ8N3QQyuauLjAf1HeLHRxZFUisSYMVPVz.GgEbuLNc72rMKmHX7pwQjQ; path=/; expires=Tue, 09-Jul-24 22:45:04 GMT; domain=.qa.sockets.stackexchange.com; HttpOnly; Secure', 'server': 'cloudflare', 'cf-ray': '8a0babac2d18bf7c-ATL'} -+-+- b'<!DOCTYPE html><html lang="en-US"><head><title>Just a moment...</title><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><meta http-equiv="X-UA-Compatible" content="IE=Edge"><meta name="robots" content="noindex,nofollow"><meta name="viewport" content="width=device-width,initial-scale=1"><style>*{box-sizing:border-box;margin:0;padding:0}html{line-height:1.15;-webkit-text-size-adjust:100%;color:#313131}button,html{font-family:system-ui,-apple-system,BlinkMacSystemFont,Segoe UI,Roboto,Helvetica Neue,Arial,Noto Sans,sans-serif,Apple Color Emoji,Segoe UI Emoji,Segoe UI Symbol,Noto Color Emoji}@media (prefers-color-scheme:dark){body{background-color:#222;color:#d9d9d9}body a{color:#fff}body a:hover{color:#ee730a;text-decoration:underline}body .lds-ring div{border-color:#999 transpar' - goodbye
True

Answer 1

Thanks for including the full response, it helped quickly identify the issue. We have been fighting a botnet that has been creating a large amount of fake accounts, unfortunately the code your team is running was also caught by one of the countermeasures that was implemented yesterday.

Edit: We found a better solution than the original fix that doesn't need any changes on your end and have pushed it out. Your code should not encounter any blocks now.