11

When running the Global Flag Summary userscript, 57 out of the 244 sites it calls to were successful, and the rest produced error status codes. Most produced 404 status codes, but towards the end of running the script, the response code changed from 404 to rate-limited. When I manually tried to access about 20 of the links that were listed as "404"s, they all worked fine.

In addition to the above, running this script also logged me out network-wide and gave me a Cloudfare challenge when I tried to access SE again.

Here is a screenshot of a small section of the errors:

A list of sites indicating indicating failure to load data for a number of sites with error code 404, and one line that says rate-limited.

Stack Exchange recently added 3 new rate limits to "prevent unauthorized access to the network," and these changes may be the cause of the new 404/rate-limiting errors.

Could this please be fixed?

Improve this question
21
  • 4
    How many requests do this script perform? How long does it take?
    – A.L
    Commented Oct 11 at 16:04
  • @A.L for me, since I regularly hang out in Charcoal HQ and flag spam on a bunch of network sites, it is looking for about 244 flag pages (and hence 244 requests). For me it usually takes a few minutes. Definetly over 10 requests per minute.
    – Starship
    Commented Oct 11 at 16:33
  • I would echo A.L's comment. Looking at the implementation, it does appear to include rate limit detection and backoff, but it may not be aligned with the new rate limit. The Meta post introducing the new limits say that scripts will be limited to 60 requests per second. Is this script exceeding 60 requests per second?
    – Thomas Owens
    Commented Oct 11 at 16:34
  • @ThomasOwens almost certainly not, unless there are a ton of requests it makes that I don't know about/understand. Floern (script's creator) has been on Meta this week, maybe they'll see this post and be able to provide a better answer...
    – Starship
    Commented Oct 11 at 16:37
  • The workaround is to register for all or most SE sites...
    – Sonic the Anonymous Hedgehog
    Commented Oct 11 at 18:25
  • @SonictheAnonymousHedgehog I think I honestly have gotten to most at this point. But why would that help? It's giving 404s on sites I have registered for.
    – Starship
    Commented Oct 11 at 18:26
  • I cannot reproduce. I only got a 404 on Math Overflow and its Meta but I went to the site, logged in, and then reloaded the global flag summary and it worked. I have 152 accounts network wide (so, 152 main + 152 meta = 304 requests), so its certainly not less than you which might avoid rate limiting.
    – VLAZ
    Commented Oct 11 at 18:45
  • 4
    If you're reporting that you got logged out during script execution, then it's not really a surprise you started getting 404s. The user flag summary page is only visible if you're logged in. And yeah, as Josh says those would have turned into 429s pretty fast because the script doesn't expect you to get logged out partway through execution, so it wouldn't have had any reason to back off. It then isn't too surprising other people can't reproduce this. We may not even be able to reproduce it ourselves - not sure on this point.
    – Slate StaffMod
    Commented Oct 11 at 18:57
  • 5
    I can see in the traffic history that we got requests from you for user flag histories on a bunch of sites, then it cuts off, and we immediately start returning 404s from your IP address for a logged-out user. I think this is a smoking gun that you got logged out partway through script execution. What logged you out, I don't know. We don't have any automatic logout triggers in the application, at least that I'm aware of. That would be very weird UX. So I suspect either you performed an action that logged you out, or there is a bug unrelated to ratelimiting.
    – Slate StaffMod
    Commented Oct 11 at 18:58
  • @Slate wow you guys can see the history anytime I view anything on SE...wow...I tested this a few times and each and every time it logged me out network-wide.
    – Starship
    Commented Oct 11 at 19:06
  • 2
    @Slate no automatic logout triggers? That can't be right. We've observed that a userscript that ends up making a lot of requests to the network search seems to kill the user session and log you out of the network. This was repeatable. The userscript got patched to only do a search on demand rather than automatically to stop the logout issues. Also, to clarify - that was few months back, not related to the new changes. But may have been related to CloudFlare.
    – VLAZ
    Commented Oct 11 at 19:08
  • With that said, I tried the Global Flag Statistics 4 times in parallel - opened 4 tabs to the global flags page. And each finished the 304 requests it does successfully. It too a bit longer but that was to be expected.
    – VLAZ
    Commented Oct 11 at 19:10
  • @VLAZ Strange then that it failed for me. Maybe try again (I was able to reproduce this 10 times in a row, with varying sites failing, and only getting the actual rate-limit sometimes)
    – Starship
    Commented Oct 11 at 19:12
  • 1
    Might I ask why you think this wouldn’t potentially happen to someone else in the future @SonictheAnonymousHedgehog
    – Starship
    Commented Oct 13 at 0:13
  • 3
    @SonictheAnonymousHedgehog I think that it can be assumed that I am not the only person that will ever experience this issue, as issues usually affect many people, unless there is reason to believe otherwise. Every bug report isn't required to have multiple people say "think happened" for it to be considered valid.
    – Starship
    Commented Oct 13 at 1:05

1 Answer 1

Reset to default
11

If you can provide me with a RayID I can verify further but I'm reasonably confident it's not the new changes that came into effect Monday.

The change made on Monday only targets things that are running out of Microsoft Azure. This is the number of times the rule that went into effect Monday, October 7th has triggered: enter image description here

The rule you've hit rate limits the source if they trigger an excessive number of 404 responses. This rate limit has been in place for at least 8 months if not since the beginning of the Cloudflare migration last year. The rationale behind the rule is that no single source should be requesting pages that don't exist at a high rate. Many malicious bots use this technique to probe the site.

Edit:

Further investigation shows that you were logged out in the middle of the script running which resulted in the 404's. The CDN does not have the ability to log users out.

Improve this answer
6
  • Might I ask what a RayId is and how I could find it?
    – Starship
    Commented Oct 11 at 18:15
  • 1
    @Starship last section of this meta.stackexchange.com/a/403462/784098
    – Josh Zhang StaffMod
    Commented Oct 11 at 18:19
  • 2
    But question is why it gave 404 for pages that actually exist? Ah, it was because OP got logged out while the script was running, so is there auto-logout new feature that is triggered when requesting X pages per minute, perhaps?
    – Shadow Wizard
    Commented Oct 11 at 19:07
  • 2
    Looking through the logs w/ @Slate it does look like the cause was a log out event during the script. The CDN does not have the ability to log users out.
    – Josh Zhang StaffMod
    Commented Oct 11 at 19:27
  • 2
    So why was I being logged out, network-wide (even on sites it never tried to access because it was rate-limited) @JoshZhang
    – Starship
    Commented Oct 11 at 19:51
  • 2
    @Starship I'm not sure however when investigating this, I was told that getting logged out was a long running known bug. That's would be on the app side which I'm not as knowledgeable on.
    – Josh Zhang StaffMod
    Commented Oct 12 at 2:46

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .