88

I'm seeing the user agent Mozilla/5.0 (compatible; stackexchangebot/1.0; +http://meta.stackexchange.com/q/130398) in my web server logs.

Why is Stack Exchange crawling my site?

Where do I report any issues?

5
  • Stack Exchange cares about links. Awesome. Now removing rel=nofollow would be the next logical step (since that obviously destroys linking as far as search engines are concerned). May 14, 2012 at 19:55
  • 1
    @konrad that was done a while ago May 14, 2012 at 21:10
  • @Jeff Oh, never mind me then. Move along. Kudos on the fine-graned approach though, I’m guessing the technical details aren’t exactly trivial. May 15, 2012 at 8:53
  • 11
    Wait a second. The number 130398 hasn't been changed since the question was posted. So you timed it perfectly so that this would be post #130398, and then set up the bot to link to this question? Jul 26, 2012 at 11:30
  • 7
    @Mechanicalsnail, that's what we call the "grace period".
    – Arjan
    Nov 18, 2012 at 20:51

2 Answers 2

103

Stack Exchange runs a bot that validates all the external links in our network of sites. This bot performs HEAD requests in a heavily throttled way. This bot was created to combat link rot in our network.

Links are tested once every 3 months.

If you notice any issues with this bot please email team@stackexchange.com


The proposed operational mechanism (this is not final as of April 26, 2012) is:

all being planned at the moment, in a nutshell, community will add a "special comment" to the post asking the creator to fix it, it will also add it to a list in /review and we will remove one of the tabs ... also we will add a couple of badges to help drive the fixing

namely:

  • Adds a comment to the post asking the author to fix it:

    "We detected an invalid link in your post, please correct it. (this message will be automatically removed when the link is fixed)"

  • Adds the post (question or answer) to a special list on the /review tab
25
  • not fully deployed yet, in initial planning phases at the moment
    – waffles
    Apr 26, 2012 at 6:41
  • 9
    What does it do when it finds a dead link? Is it flagged at least? It would be awesome if posts with dead links appeared in the review page (or 10k). Apr 26, 2012 at 7:00
  • 17
    @JeffMercado all being planned at the moment, in a nutshell, community will add a "special comment" to the post asking the creator to fix it, it will also add it to a list in /review and we will remove one of the tabs ... also we will add a couple of badges to help drive the fixing
    – waffles
    Apr 26, 2012 at 7:03
  • More stinking badgers? W00t! Apr 26, 2012 at 10:49
  • 4
    I hope there will be badges for fixing link in your own posts, and that the auto added comment will be auto removed once the link has been fixed. It may be worth only adding items to the review tab, if the author has not fixed the link themselves within n days, also should the review tab only show posts with lots of views that have broken links, so the most important post are fixed first? Apr 26, 2012 at 11:30
  • 18
    Can you make this flag / remove known URL shorteners since this pretty much kills the existing "too expensive" argument.
    – Flexo
    Apr 26, 2012 at 12:38
  • 1
    @AviD actually you can sometimes find the important content, I saved a linkrot post thanks to the Wayback MAchine the other day.
    – Ben Brocka
    Apr 26, 2012 at 13:36
  • 2
    @waffles So I should deliberately add a dead link, wait 3 months and then edit it to get the badge? ;) Apr 26, 2012 at 23:36
  • 4
    @yoda I think a better approach is to add a good link, then launch a DDOS on the target site, take it down, forever, then replace the link with a link to the wayback machine. AND PROFIT. I hear botnets can be rented for quite cheap these days.
    – waffles
    Apr 26, 2012 at 23:45
  • 3
    Curious: are you also doing this for images? And if adding @awoodland's URL shorteners, then maybe personal storage such as Dropbox and the like could get some attention too. (Especially for images, I feel.) (Nice, by he way!)
    – Arjan
    Apr 27, 2012 at 7:13
  • 3
    Is this still active? meta.stackoverflow.com/review/broken-links is no more and Google only finds some hits on Meta.
    – Arjan
    Nov 18, 2012 at 20:46
  • 1
    @sixlettervariables I'm not sure, but I'll check.
    – casperOne
    Mar 11, 2013 at 19:40
  • 1
    @TimPost Is this status-completed ? (The bot)
    – asheeshr
    Apr 13, 2013 at 12:20
  • 1
    @TimPost I have to ask, which Monday ? ;)
    – asheeshr
    May 9, 2013 at 12:41
  • 2
    @TimPost FYI on the wayback solution I just posted, I always disliked nagging users about this; now we don't have to (as much). Sep 26, 2013 at 0:27
33

The Wayback Machine now has an availability API

http://archive.org/about/wayback_api.php

It might make sense to incorporate this into the crawl. If we can't find the original source link in the crawl, ask Wayback if they have a copy of said link, and if they do we can fix the post ourselves without bugging users about it.

The crawl is of course still necessary to check the validity of the link. But we have a much better fallback that can save everyone a lot of work, and reduce load on the human-powered queues. Robot work is always preferable to human work..

7
  • 6
    Oh, now that's cool! Sep 26, 2013 at 1:29
  • 1
    This is awesome. Many users fix broken links by replacing them with a link to the Wayback copy since it's the simplest thing to do, hooray for robots! Sticking this on our feature agenda.
    – Tim Post
    Sep 26, 2013 at 5:17
  • 3
    This is going to the devs sometime today to look into implementing. It should fix the immediate problem, provided that the wayback copy is the same age or newer than the post when the link was made, and we only automatically re-write them after several failed checks. There will be a unique PostHistory entry when this happens, so we can conceivably find / improve these posts if they too heavily depend on the now fixed link (but later, another project, another time). Hopefully nothing will get in the way of it working.
    – Tim Post
    Nov 22, 2013 at 15:31
  • 2
    @TimPost any updates? Mar 23, 2014 at 7:50
  • 1
    @JeffAtwood It went on the core call, but there's a bit of resistance. The Wayback API gives us everything we need to know in order to responsibly re-write the link (e.g. we're not linking to a version much older than what the author was originally linking to that never updated) - but the Community user would be making a ton of edits. Even though I specified a PostHistory event for Community fixing a link so these could be reviewed if needed, we'd need to better address the 'if needed' before this could go in. (1/2)
    – Tim Post
    Mar 24, 2014 at 4:31
  • 1
    The other concern was that this would (at least in part) prevent posts that depended too heavily on a link from being surfaced and fixed to be fundamentally better posts, I don't entirely agree with that. Anyway, it's all leading back to finally getting the link review queue in shape to actually be used, at which time I'll propose this again in the context of actually having that in place - where we put the 'fixed' posts in front of people for edits to make them fundamentally better as is possible. (2/2)
    – Tim Post
    Mar 24, 2014 at 4:35
  • 1
    @TimPost an alternative would be something like the IA 404 handler but on SE's side. As you rewrite the Amazon links in the output, you may rewrite certain links to go to a handler which checks for the existence of the target or redirects to archive.org.
    – Nemo
    Oct 11, 2015 at 12:00

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .