What is meta? ×
Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 130 Stack Exchange communities.

I'm seeing the user agent Mozilla/5.0 (compatible; stackexchangebot/1.0; +http://meta.stackexchange.com/q/130398) in my web server logs.

Why is Stack Exchange crawling my site?

Where do I report any issues?

share|improve this question
    
Stack Exchange cares about links. Awesome. Now removing rel=nofollow would be the next logical step (since that obviously destroys linking as far as search engines are concerned). –  Konrad Rudolph May 14 '12 at 19:55
1  
@konrad that was done a while ago –  Jeff Atwood May 14 '12 at 21:10
    
@Jeff Oh, never mind me then. Move along. Kudos on the fine-graned approach though, I’m guessing the technical details aren’t exactly trivial. –  Konrad Rudolph May 15 '12 at 8:53
9  
Wait a second. The number 130398 hasn't been changed since the question was posted. So you timed it perfectly so that this would be post #130398, and then set up the bot to link to this question? –  Mechanical snail Jul 26 '12 at 11:30
5  
@Mechanicalsnail, that's what we call the "grace period". –  Arjan Nov 18 '12 at 20:51

2 Answers 2

Stack Exchange runs a bot that validates all the external links in our network of sites. This bot performs HEAD requests in a heavily throttled way. This bot was created to combat link rot in our network.

Links are tested once every 3 months.

If you notice any issues with this bot please email team@stackexchange.com


The proposed operational mechanism (this is not final as of April 26, 2012) is:

all being planned at the moment, in a nutshell, community will add a "special comment" to the post asking the creator to fix it, it will also add it to a list in /review and we will remove one of the tabs ... also we will add a couple of badges to help drive the fixing

namely:

  • Adds a comment to the post asking the author to fix it:

    "We detected an invalid link in your post, please correct it. (this message will be automatically removed when the link is fixed)"

  • Adds the post (question or answer) to a special list on the /review tab
share|improve this answer
    
not fully deployed yet, in initial planning phases at the moment –  waffles Apr 26 '12 at 6:41
8  
What does it do when it finds a dead link? Is it flagged at least? It would be awesome if posts with dead links appeared in the review page (or 10k). –  Jeff Mercado Apr 26 '12 at 7:00
17  
@JeffMercado all being planned at the moment, in a nutshell, community will add a "special comment" to the post asking the creator to fix it, it will also add it to a list in /review and we will remove one of the tabs ... also we will add a couple of badges to help drive the fixing –  waffles Apr 26 '12 at 7:03
    
More stinking badgers? W00t! –  Donal Fellows Apr 26 '12 at 10:49
4  
I hope there will be badges for fixing link in your own posts, and that the auto added comment will be auto removed once the link has been fixed. It may be worth only adding items to the review tab, if the author has not fixed the link themselves within n days, also should the review tab only show posts with lots of views that have broken links, so the most important post are fixed first? –  Ian Ringrose Apr 26 '12 at 11:30
    
I see answers only referring to external links all the time. This is something that should be fighted more vigorously on the moderation side. But this addition is also welcome! –  UncleZeiv Apr 26 '12 at 11:34
    
@UncleZeiv How are moderators supposed to fight it? We aren't given any information on this except when users point out where there's link rot, in which case, we go and address it. –  casperOne Apr 26 '12 at 12:21
17  
Can you make this flag / remove known URL shorteners since this pretty much kills the existing "too expensive" argument. –  Flexo Apr 26 '12 at 12:38
    
@AviD actually you can sometimes find the important content, I saved a linkrot post thanks to the Wayback MAchine the other day. –  Ben Brocka Apr 26 '12 at 13:36
2  
@waffles So I should deliberately add a dead link, wait 3 months and then edit it to get the badge? ;) –  Lorem Ipsum Apr 26 '12 at 23:36
3  
@yoda I think a better approach is to add a good link, then launch a DDOS on the target site, take it down, forever, then replace the link with a link to the wayback machine. AND PROFIT. I hear botnets can be rented for quite cheap these days. –  waffles Apr 26 '12 at 23:45
    
@awoodland I do not really intend to walk through comments though, the content I care about most here is post bodies –  waffles Apr 26 '12 at 23:47
3  
Curious: are you also doing this for images? And if adding @awoodland's URL shorteners, then maybe personal storage such as Dropbox and the like could get some attention too. (Especially for images, I feel.) (Nice, by he way!) –  Arjan Apr 27 '12 at 7:13
1  
@sixlettervariables I'm not sure, but I'll check. –  casperOne Mar 11 '13 at 19:40
1  
@TimPost FYI on the wayback solution I just posted, I always disliked nagging users about this; now we don't have to (as much). –  Jeff Atwood Sep 26 '13 at 0:27

The Wayback Machine now has an availability API

http://archive.org/about/wayback_api.php

It might make sense to incorporate this into the crawl. If we can't find the original source link in the crawl, ask Wayback if they have a copy of said link, and if they do we can fix the post ourselves without bugging users about it.

The crawl is of course still necessary to check the validity of the link. But we have a much better fallback that can save everyone a lot of work, and reduce load on the human-powered queues. Robot work is always preferable to human work..

share|improve this answer
5  
Oh, now that's cool! –  Andrew Barber Sep 26 '13 at 1:29
    
This is awesome. Many users fix broken links by replacing them with a link to the Wayback copy since it's the simplest thing to do, hooray for robots! Sticking this on our feature agenda. –  Tim Post Sep 26 '13 at 5:17
2  
This is going to the devs sometime today to look into implementing. It should fix the immediate problem, provided that the wayback copy is the same age or newer than the post when the link was made, and we only automatically re-write them after several failed checks. There will be a unique PostHistory entry when this happens, so we can conceivably find / improve these posts if they too heavily depend on the now fixed link (but later, another project, another time). Hopefully nothing will get in the way of it working. –  Tim Post Nov 22 '13 at 15:31
1  
@TimPost any updates? –  Jeff Atwood Mar 23 at 7:50
    
@JeffAtwood It went on the core call, but there's a bit of resistance. The Wayback API gives us everything we need to know in order to responsibly re-write the link (e.g. we're not linking to a version much older than what the author was originally linking to that never updated) - but the Community user would be making a ton of edits. Even though I specified a PostHistory event for Community fixing a link so these could be reviewed if needed, we'd need to better address the 'if needed' before this could go in. (1/2) –  Tim Post Mar 24 at 4:31
    
The other concern was that this would (at least in part) prevent posts that depended too heavily on a link from being surfaced and fixed to be fundamentally better posts, I don't entirely agree with that. Anyway, it's all leading back to finally getting the link review queue in shape to actually be used, at which time I'll propose this again in the context of actually having that in place - where we put the 'fixed' posts in front of people for edits to make them fundamentally better as is possible. (2/2) –  Tim Post Mar 24 at 4:35

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .