21

Here is the conversation on Twitter:

Jeff Atwood (codinghorror):

oh look, Yahoo Pipes doesn't respect robots.txt. I am SHOCKED! (ip block ban time)

Simon Willison (simonw):

@codinghorror should RSS consumers obey robots.txt? Pipes doesn't crawl, it's directed-if you robots-off aggregator, why publish RSS at all?

Jeff Atwood (codinghorror):

r @simonw Pipes is crawling HTML when we explicitly told it not to in robots.txt. Verified logs. Result: perma-ban from all IP ranges.

To me, it sounds like Jeff doesn't understand that Yahoo! Pipes isn't a robot and therefore has no need to obey robots.txt and that Yahoo! Pipes is for more than feeds - they have specific "modules" designed to retrieve XML, JSON, and HTML (all offered by StackOverflow).

Was it really eating that many resources? I know a new server was installed for GoogleBot to have fun with, since it wanted to index gigs of stuff from SO. If it was, I can understand blocking it - it's far more important to allow your users to have a good, unhampered experience and then to allow indexers to play with your site (as far as I know, Pipes does not index anything).

I was actually playing with Yahoo! Pipes (since Google doesn't have anything like it that I know of) to try to come up with some cool SO feeds that I can make public, but also parse for cool data, so is there an explanation other than "it doesn't obey rules that I think it should"?

Improve this question
9
  • 1
    Made a slight change to your title, since none of us are really qualified to explain absolutely why Jeff does this or that :)
    – Sampson
    Sep 29, 2009 at 12:28
  • 1
    Good call. I like it. Luckily, I had only begun playing with Yahoo! Pipes and had about 1 hour invested in it...if I had more, I would be all tripping up in Jeff's face. :P
    – Thomas Owens
    Sep 29, 2009 at 12:31
  • 11
    I don't know what Yahoo! Pipes is, but I know I would have liked it better if it were called Yahoo! Tubes.
    – Eric
    Sep 29, 2009 at 12:31
  • 2
    Eric: Pipes are like a subset of the Tubes...it's like the tubes are what brings water to your house, and then the pipes bring it to different places. It's really how Yahoo! Pipes work. You bring things from the InterTubes and do stuff with it in different places.
    – Thomas Owens
    Sep 29, 2009 at 12:33
  • Do you have to wash your hands after you've been playing with the tubes? The pipes that is?
    – random
    Sep 29, 2009 at 12:36
  • 1
    With the new Yahoo! campain would it have been You!Tubes then?
    – Ivo Flipse
    Sep 29, 2009 at 13:05
  • 3
    Never heard of Yahoo Pipes before, but if it's really the bandwidth hog that people claim it is, I would advise all administrators to do a complete IP block for Yahoo. Crawling HTML is real bad.
    – Wim ten Brink
    Sep 29, 2009 at 13:57
  • 10
    @Workshop Alex, I'm confused. If you've "Never heard of Yahoo Pipes before," how exactly did you hear that it's a "bandwidth hog"?
    – Sampson
    Sep 29, 2009 at 15:19
  • 5
    I bet he read it on Twitter. If it's on Twitter, it must be true, right?
    – Shog9
    Sep 29, 2009 at 16:26

5 Answers 5

Reset to default
18

Jeff doesn't understand that Yahoo! Pipes isn't a robot and therefore has no need to obey robots.txt

Really? That's funny, because that's not what the Yahoo! Pipes documentation says.

How do I keep Pipes from accessing my web pages?

There are three ways you can prevent your page content from being used within a Yahoo! Pipe:

3) Using robots.txt

User-agent: Yahoo Pipes 1.0
Disallow: /

Too bad it LIES!

I don't really have a problem with Pipes using the RSS, but Pipes not respecting robots.txt for html requests is a dealbreaker.

Update: Talking at London DevDays to Christian Heilmann who is a Yahoo dev evangelist and works on the YQL team. We're discussing removing the Yahoo Pipes block and working on other types of yahoo YQL integration; done right it could be a form of API.

Improve this answer
5
  • 2
    How much bandwidth was it actually eating, though? And are you sure you weren't being eaten up by Yahoo Pipes 2.0 (used by YQL), which is totally different than Yahoo Pipes 1.0 (used by Pipes)? And is there some way to have a rate-limited Pipe access to the SO pages? Does your block extend to feeds and JSON/XML content or just HTML pages?
    – Thomas Owens
    Sep 29, 2009 at 17:11
  • 8
    It's a matter of respect. Jeff respects robots. I respect robots. Flight of the Conchords respects robots. Yahoo! Pipes does not. So they get no respect in return.
    – devinb
    Sep 29, 2009 at 19:05
  • 2
    Have you contacted Yahoo about this? Quoted from the link you mention, "Because Pipes is not a web crawler (the service only retrieves URLs when requested to by a Pipe author or user) Pipes does not follow the robots exclusion protocol (when fetching feeds), and won't check your robots.txt file"
    – dbr
    Oct 24, 2009 at 14:24
  • 3
    suggestions.yahoo.com/detail/?prop=Pipes&fid=75361 "The "Fetch Page" module currently honors the robots.txt file" - it ignoring your robots.txt file may be a bug they are unaware of..
    – dbr
    Oct 24, 2009 at 14:30
  • is this still not working? I really would like to use pipes now (3 years later), but it comes empty :(
    – Felipe Hoffa
    Jun 21, 2013 at 18:30
7

So... you want Jeff to say "I was wrong, Pipes shouldn't have to obey robots.txt, but it's still sucking down way too many resources so it's still blocked."? What would that accomplish?

alt text

Improve this answer
5
  • 6
    If Jeff doesn't accept that he's incorrect, other people will undoubtedly circulate the same incorrect information, citing his stance as evidence to back up their mistake.
    – NickFitz
    Sep 29, 2009 at 15:54
  • 2
    Truth is always a worthwhile accomplishment.
    – Lance Roberts
    Sep 29, 2009 at 16:22
  • 5
    YOUR FIRST DUTY IS TO THE TRUTH! Whether it's SCIENTIFIC TRUTH, or HISTORICAL TRUTH, or PERSONAL TRUTH!
    – Thomas Owens
    Sep 29, 2009 at 16:36
  • And it would accomplish one of two things - either Pipes gets unblocked because it wasn't eating too many resources or Pipes stays blocked and web developers learn exactly how hungry Pipes is.
    – Thomas Owens
    Sep 29, 2009 at 16:39
  • 1
    @www.nickfitz.co.uk: Are you sure Jeff is incorrect? He's the guy who's looked at the logs. I'd take his word over what Pipes is doing on his site rather than somebody else's word about what it really should be doing.
    – David Thornley
    Oct 28, 2009 at 14:02
3

I suspect that Jeff (who has access to the log) knows more than about what Yahoo is doing to his site than sinmonw does, and I'm certain that it is more reasonable for a site owner to decide who (s)he expects to obey robots.txt than the operator of the robots.

End of story.

Improve this answer
5
  • 6
    Jeff is indeed at liberty to block whomever and whatever he wishes. However, he cannot justify this block on the basis that Pipes should observe robots.txt, any more than he could justify blocking you or I on such a basis. Pipes is not a robot, so robots.txt does not, by specification, apply to it.
    – NickFitz
    Sep 29, 2009 at 15:31
  • 1
    You miss the point. The robots.txt is there to let Jeff say who may or may not engage in automated downloading. Yahoo pipes is a automation system. DO you really think it would be OK for me to write IAmNotARobot.sh and set it loose not obeying robots.txt?
    – dmckee --- ex-moderator kitten
    Sep 29, 2009 at 18:03
  • @dmckee: That sounds perfectly acceptable to me.
    – devinb
    Sep 29, 2009 at 19:06
  • Your web browser (if you are using one) is an automated system. I agree completely with www.nickfitz.co.uk.
    – user148743
    Jul 12, 2010 at 22:38
  • @user: my browser downloads files at an average pace throttled by my ability to scan the content to see if I am interested and consume some of it in its entirety. This is markedly different from a system which can process files as fast as they can be downloaded. That is why robots.txt exists: to let people off content that they do not want slurped wholesale.
    – dmckee --- ex-moderator kitten
    Jul 13, 2010 at 4:31
2

Huh.

I've only used Pipes a couple of times, and wasn't relying on it for anything regularly... Still, it was rather fun to play around with it, massaging the various SO-family Atom feeds into new and interesting shapes.

Oh well... at least Google Reader is still available. ALL HAIL OUR GOOGLY OVERLORDS!

Improve this answer
2
  • HAIL GOOGLE! :-) Indeed, I use the Google reader a lot, to check upon the new messages in SO. Too bad there are about 1,000 new SO messages every day but still...
    – Wim ten Brink
    Sep 29, 2009 at 15:19
  • If Y! pipes were unblocked you would have a way to filter them. One that wouldn't cause a net increase in traffic to SO. I don't understand the reasoning behind the block.
    – pguardiario
    Nov 4, 2011 at 11:31
0

I currently use Yahoo! Pipes to correct problems using the RSS feed from StackOverflow. In particular, I use twitterfeed to feed the RSS to twitter/facebook and the relative URLs are not transformed to absolute URLs, thus people tend to complain about the links. To correct this, I replace the leading / with http://stackoverflow.com/.

I guess I could run the StackOverflow feed through Google Reader, share it and then use the shared feed in Yahoo! Pipes?

Improve this answer
4
  • Looks like it just got unblocked?
    – Cade Roux
    Sep 30, 2009 at 0:05
  • Apparently intermittent - I guess they use multiple IPs
    – Cade Roux
    Sep 30, 2009 at 0:31
  • Or the pipes now show robots.txt some respect.
    – AnonJr
    Sep 30, 2009 at 1:21
  • Looks unblocked again.
    – Cade Roux
    Sep 30, 2009 at 19:43

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .