I'm seeing the user agent
Mozilla/5.0 (compatible; stackexchangebot/1.0; +http://meta.stackexchange.com/q/130398) in my web server logs.
Why is Stack Exchange crawling my site?
Where do I report any issues?
Stack Exchange runs a bot that validates all the external links in our network of sites. This bot performs
Links are tested once every 3 months.
If you notice any issues with this bot please email
The proposed operational mechanism (this is not final as of April 26, 2012) is:
|show 10 more comments|
The Wayback Machine now has an availability API
It might make sense to incorporate this into the crawl. If we can't find the original source link in the crawl, ask Wayback if they have a copy of said link, and if they do we can fix the post ourselves without bugging users about it.
The crawl is of course still necessary to check the validity of the link. But we have a much better fallback that can save everyone a lot of work, and reduce load on the human-powered queues. Robot work is always preferable to human work..