What is meta? ×
Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 131 Stack Exchange communities.

Here is the conversation on Twitter:

Jeff Atwood (codinghorror):

oh look, Yahoo Pipes doesn't respect robots.txt. I am SHOCKED! (ip block ban time)

Simon Willison (simonw):

@codinghorror should RSS consumers obey robots.txt? Pipes doesn't crawl, it's directed-if you robots-off aggregator, why publish RSS at all?

Jeff Atwood (codinghorror):

r @simonw Pipes is crawling HTML when we explicitly told it not to in robots.txt. Verified logs. Result: perma-ban from all IP ranges.

To me, it sounds like Jeff doesn't understand that Yahoo! Pipes isn't a robot and therefore has no need to obey robots.txt and that Yahoo! Pipes is for more than feeds - they have specific "modules" designed to retrieve XML, JSON, and HTML (all offered by StackOverflow).

Was it really eating that many resources? I know a new server was installed for GoogleBot to have fun with, since it wanted to index gigs of stuff from SO. If it was, I can understand blocking it - it's far more important to allow your users to have a good, unhampered experience and then to allow indexers to play with your site (as far as I know, Pipes does not index anything).

I was actually playing with Yahoo! Pipes (since Google doesn't have anything like it that I know of) to try to come up with some cool SO feeds that I can make public, but also parse for cool data, so is there an explanation other than "it doesn't obey rules that I think it should"?

share|improve this question
Made a slight change to your title, since none of us are really qualified to explain absolutely why Jeff does this or that :) –  Jonathan Sampson Sep 29 '09 at 12:28
Good call. I like it. Luckily, I had only begun playing with Yahoo! Pipes and had about 1 hour invested in it...if I had more, I would be all tripping up in Jeff's face. :P –  Thomas Owens Sep 29 '09 at 12:31
I don't know what Yahoo! Pipes is, but I know I would have liked it better if it were called Yahoo! Tubes. –  Eric Sep 29 '09 at 12:31
Eric: Pipes are like a subset of the Tubes...it's like the tubes are what brings water to your house, and then the pipes bring it to different places. It's really how Yahoo! Pipes work. You bring things from the InterTubes and do stuff with it in different places. –  Thomas Owens Sep 29 '09 at 12:33
Do you have to wash your hands after you've been playing with the tubes? The pipes that is? –  random Sep 29 '09 at 12:36
With the new Yahoo! campain would it have been You!Tubes then? –  Ivo Flipse Sep 29 '09 at 13:05
Never heard of Yahoo Pipes before, but if it's really the bandwidth hog that people claim it is, I would advise all administrators to do a complete IP block for Yahoo. Crawling HTML is real bad. –  Wim ten Brink Sep 29 '09 at 13:57
@Workshop Alex, I'm confused. If you've "Never heard of Yahoo Pipes before," how exactly did you hear that it's a "bandwidth hog"? –  Jonathan Sampson Sep 29 '09 at 15:19
I bet he read it on Twitter. If it's on Twitter, it must be true, right? –  Shog9 Sep 29 '09 at 16:26

5 Answers 5

up vote 18 down vote accepted

Jeff doesn't understand that Yahoo! Pipes isn't a robot and therefore has no need to obey robots.txt

Really? That's funny, because that's not what the Yahoo! Pipes documentation says.

How do I keep Pipes from accessing my web pages?

There are three ways you can prevent your page content from being used within a Yahoo! Pipe:

3) Using robots.txt

User-agent: Yahoo Pipes 1.0
Disallow: /

Too bad it LIES!

I don't really have a problem with Pipes using the RSS, but Pipes not respecting robots.txt for html requests is a dealbreaker.

Update: Talking at London DevDays to Christian Heilmann who is a Yahoo dev evangelist and works on the YQL team. We're discussing removing the Yahoo Pipes block and working on other types of yahoo YQL integration; done right it could be a form of API.

share|improve this answer
How much bandwidth was it actually eating, though? And are you sure you weren't being eaten up by Yahoo Pipes 2.0 (used by YQL), which is totally different than Yahoo Pipes 1.0 (used by Pipes)? And is there some way to have a rate-limited Pipe access to the SO pages? Does your block extend to feeds and JSON/XML content or just HTML pages? –  Thomas Owens Sep 29 '09 at 17:11
It's a matter of respect. Jeff respects robots. I respect robots. Flight of the Conchords respects robots. Yahoo! Pipes does not. So they get no respect in return. –  devinb Sep 29 '09 at 19:05
Have you contacted Yahoo about this? Quoted from the link you mention, "Because Pipes is not a web crawler (the service only retrieves URLs when requested to by a Pipe author or user) Pipes does not follow the robots exclusion protocol (when fetching feeds), and won't check your robots.txt file" –  dbr Oct 24 '09 at 14:24
suggestions.yahoo.com/detail/?prop=Pipes&fid=75361 "The "Fetch Page" module currently honors the robots.txt file" - it ignoring your robots.txt file may be a bug they are unaware of.. –  dbr Oct 24 '09 at 14:30
is this still not working? I really would like to use pipes now (3 years later), but it comes empty :( –  Fh. Jun 21 '13 at 18:30

So... you want Jeff to say "I was wrong, Pipes shouldn't have to obey robots.txt, but it's still sucking down way too many resources so it's still blocked."? What would that accomplish?

alt text

share|improve this answer
If Jeff doesn't accept that he's incorrect, other people will undoubtedly circulate the same incorrect information, citing his stance as evidence to back up their mistake. –  NickFitz Sep 29 '09 at 15:54
Truth is always a worthwhile accomplishment. –  Lance Roberts Sep 29 '09 at 16:22
And it would accomplish one of two things - either Pipes gets unblocked because it wasn't eating too many resources or Pipes stays blocked and web developers learn exactly how hungry Pipes is. –  Thomas Owens Sep 29 '09 at 16:39
@www.nickfitz.co.uk: Are you sure Jeff is incorrect? He's the guy who's looked at the logs. I'd take his word over what Pipes is doing on his site rather than somebody else's word about what it really should be doing. –  David Thornley Oct 28 '09 at 14:02

I suspect that Jeff (who has access to the log) knows more than about what Yahoo is doing to his site than sinmonw does, and I'm certain that it is more reasonable for a site owner to decide who (s)he expects to obey robots.txt than the operator of the robots.

End of story.

share|improve this answer
Jeff is indeed at liberty to block whomever and whatever he wishes. However, he cannot justify this block on the basis that Pipes should observe robots.txt, any more than he could justify blocking you or I on such a basis. Pipes is not a robot, so robots.txt does not, by specification, apply to it. –  NickFitz Sep 29 '09 at 15:31
You miss the point. The robots.txt is there to let Jeff say who may or may not engage in automated downloading. Yahoo pipes is a automation system. DO you really think it would be OK for me to write IAmNotARobot.sh and set it loose not obeying robots.txt? –  dmckee Sep 29 '09 at 18:03
@dmckee: That sounds perfectly acceptable to me. –  devinb Sep 29 '09 at 19:06
Your web browser (if you are using one) is an automated system. I agree completely with www.nickfitz.co.uk. –  user148743 Jul 12 '10 at 22:38
@user: my browser downloads files at an average pace throttled by my ability to scan the content to see if I am interested and consume some of it in its entirety. This is markedly different from a system which can process files as fast as they can be downloaded. That is why robots.txt exists: to let people off content that they do not want slurped wholesale. –  dmckee Jul 13 '10 at 4:31


I've only used Pipes a couple of times, and wasn't relying on it for anything regularly... Still, it was rather fun to play around with it, massaging the various SO-family Atom feeds into new and interesting shapes.

Oh well... at least Google Reader is still available. ALL HAIL OUR GOOGLY OVERLORDS!

share|improve this answer
HAIL GOOGLE! :-) Indeed, I use the Google reader a lot, to check upon the new messages in SO. Too bad there are about 1,000 new SO messages every day but still... –  Wim ten Brink Sep 29 '09 at 15:19
If Y! pipes were unblocked you would have a way to filter them. One that wouldn't cause a net increase in traffic to SO. I don't understand the reasoning behind the block. –  pguardiario Nov 4 '11 at 11:31

I currently use Yahoo! Pipes to correct problems using the RSS feed from StackOverflow. In particular, I use twitterfeed to feed the RSS to twitter/facebook and the relative URLs are not transformed to absolute URLs, thus people tend to complain about the links. To correct this, I replace the leading / with http://stackoverflow.com/.

I guess I could run the StackOverflow feed through Google Reader, share it and then use the shared feed in Yahoo! Pipes?

share|improve this answer
Looks like it just got unblocked? –  Cade Roux Sep 30 '09 at 0:05
Apparently intermittent - I guess they use multiple IPs –  Cade Roux Sep 30 '09 at 0:31
Or the pipes now show robots.txt some respect. –  AnonJr Sep 30 '09 at 1:21
Looks unblocked again. –  Cade Roux Sep 30 '09 at 19:43

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .