Since robots.txt doesn't force us to follow it:
The Robot Exclusion Standard, also known as the Robots Exclusion Protocol or robots.txt protocol, is a convention to prevent cooperating web crawlers and other web robots from accessing all or part of a website which is otherwise publicly viewable.
Basically, you want to be the good-guy cooperating crawler, right? Well, we don't crawl links. Crawlers spread out through every link they get and basically end up parsing nearly the whole site. That's bad when a site has tons of links that don't need to be crawled.
On the other hand, we take a link, point, and shoot. There's no "crawling" involved--nothing that can spiral into bigtime server load if we hit a page with millions of links. That's pretty much why we have
robots.txt-- to prevent the robots from
taking over the world overloading the site.
Though, if you notice, there's the "other web robots" in the Wikipedia snippet. Not complying with
robots.txt will mean that we are complying with the intentions of
robots.txt, but not with the actual rules. Which is OK, since overly broad rules are meant to be broken. And, it's not really a "rule" per se--it's a "listen if you know what's good for you" thingy:
Despite the use of the terms "allow" and "disallow", the protocol is purely advisory.
And, we're old enough to know what's good for us without having them tell us, right? ;-)
So just point-and-shoot. Don't even fetch robots.txt, it unecessarily increases server load a teensy bit. Though, as @Dennis suggested, you can consult robots.txt if you have a truckload of links on the same domain that you need to verify.