I'm seeing the user agent
Mozilla/5.0 (compatible; stackexchangebot/1.0; +http://meta.stackexchange.com/q/130398) in my web server logs.
Why is Stack Exchange crawling my site?
Where do I report any issues?
Meta Stack Exchange is a question and answer site for meta-discussion of the Stack Exchange family of Q&A websites. It only takes a minute to sign up.Sign up to join this community
Stack Exchange runs a bot that validates all the external links in our network of sites. This bot performs
HEAD requests in a heavily throttled way. This bot was created to combat link rot in our network.
Links are tested once every 3 months.
If you notice any issues with this bot please email
The proposed operational mechanism (this is not final as of April 26, 2012) is:
all being planned at the moment, in a nutshell, community will add a "special comment" to the post asking the creator to fix it, it will also add it to a list in /review and we will remove one of the tabs ... also we will add a couple of badges to help drive the fixing
Adds a comment to the post asking the author to fix it:
The Wayback Machine now has an availability API
It might make sense to incorporate this into the crawl. If we can't find the original source link in the crawl, ask Wayback if they have a copy of said link, and if they do we can fix the post ourselves without bugging users about it.
The crawl is of course still necessary to check the validity of the link. But we have a much better fallback that can save everyone a lot of work, and reduce load on the human-powered queues. Robot work is always preferable to human work..