20

Searching the Wayback Machine for URLs of user profile pages doesn't return some user pages, eg. my user profile pages on the SE sites that I have ever joined since August (when my account was registered.) Example:

https://web.archive.org/web/20231*/writing.stackexchange.com/*

And it is not just me; there are very few user profile pages in general being archived compared to the beginning of this year. This lack of user page archives affects all SE sites and not just Writing.

9
  • 9
    For Writing specifically, the rate of Archive.org profile saves (according to the last saved date) decreased suddenly after June 11th. I have no explanation for this. It looks like some other sites saw a similar decrease.
    – Laurel
    Commented Dec 3, 2023 at 0:59
  • 7
    Perhaps the amount of requests coming from archive.org got restricted around that time? It's especially interesting since this happens at the near-exact same time the data dumps stop.
    – Joachim
    Commented Dec 3, 2023 at 1:31
  • 3
    Prior to late September, I see a lot of snapshot attempts getting a 4xx error. It seems like their crawler is being rate limited. I'm seeing similar patterns across Writing, which is the subject of this question, but also Software Engineering and Project Management. I wonder if they prioritize certain pages to reduce rate limiting. Commented Dec 3, 2023 at 2:26
  • 8
    Based on an earlier question, it also seems like the rate limiter is network-wide. I wonder if it's a change in the Internet Archive crawler. Although I'm also wondering if this isn't a feature request to grant the Internet Archive a much more substantial rate limit so they can better do their mission. It would be more consistent with the free and open nature of the contributions. If there were any changes on SE's end in either direction, I'd consider that a key product decision that should be communicated in a timely manner. Commented Dec 3, 2023 at 2:31
  • Some people might not want their user profiles archived and might have send a request to the Internet Archive to exclude their profile but it's unlikely for so many users to do this.
    – The_AH
    Commented Dec 3, 2023 at 6:10
  • 3
    I seem to recall low-reputation users have had their profile pages marked noindex at some point. Commented Dec 3, 2023 at 9:41
  • 7
    Maybe an inadvertent effect of the Fastly -> Cloudflare change? Commented Dec 3, 2023 at 12:59
  • 1
    @user3840170 yep, that was a defense against spam. Spammers would create a profile then stuff the 'about me' section with links to spam and SEO terms to rank higher in Google.
    – Robotnik
    Commented Dec 3, 2023 at 22:51
  • 1
    @ThomasOwens Requested here: Don’t rate limit Archive.org as much. Since we haven't seen any errors like that, I'm not sure if that's the problem or not.
    – Laurel
    Commented Dec 4, 2023 at 20:34

0

You must log in to answer this question.