22

We have a search utility on Yfrog that indexes all tweets that hit yfrog.com (we have 1.5 billion tweets indexed). We want to add all Stack Overflow questions to our index so that we could refer interested users to Stack Overflow sites.

Given that the dump data is incomplete, our questions are:

  • Are there any restrictions on crawling?
  • Do we need to be whitelisted?
  • Is it possible to get a dump of all questions so that we can index the sites without spidering (crawling and discovering questions)? (A list of questions will allow us to index the content directly)

For those who are not familiar with Yfrog, we are a social media utility that aggregates personal twitter feeds, visualizes your conversations on twitter, and provides search results mine from your own feed (so that you can find all tweets you have ever posted).

We are planning to extend this search service with stackoverflow and wikipedia index.

Improve this question
3
  • 13
    For those of us who aren't familiar with Yfrog, could you describe briefly what / how this would be used, and how it would be useful to your users?
    – Shog9
    Jun 12, 2012 at 22:18
  • May I email you, please? We're seeing strange access patterns in the web logs. If this is ok, should I use the address in your profile?
    – Jarrod Dixon
    Jun 13, 2012 at 1:53
  • @JarrodDixon Yes of course.
    – Jack Levin
    Jun 13, 2012 at 5:14

1 Answer 1

Reset to default
20

In short we expect that when you crawl, you follow the basic rules of being a good spider. You can find a good introduction here. So make sure you do what that article says.

Besides or in addition to that, in summary the most import things are:

  • Use compression (request that the content be compressed)
  • Respect our robots.txt
  • Crawl at a rate that is reasonably proportional to the traffic you give us. For example, we are okay with Google sucking a lot of our bandwidth because we get a lot of traffic from them. I know you can't be sure of what this is, but use common sense to guess what it is.
  • We have a rate limiter. Pay attention if you start getting 503s from us and back off if you do. If you feel that the rate limiter is too restrictive we could possibly white-list you when the time comes.
  • Have a User agent string that lets us know how to contact you if we are unhappy with an aspect of how you are crawling us.

If crawlers don't follow these basic rules you are likely to get blocked by us. Frankly this is all off the top of my head (usually, people don't ask first) so I would recommend you follow up and check this answer for any additional edits before you start crawling us.

Besides all of this, asking on meta before you start crawling us goes above and beyond what most people who end up getting blocked do, so we will try to make an effort to return the courtesy.

Improve this answer
8
  • Thanks, is it easy to generate a list of URIs with questions? We can significantly limit the scope and therefore load for all concerned. Would it be a good idea to give you our IPs to get whitelisted, assuming we follow robots.txt and crawl slow?
    – Jack Levin
    Jun 13, 2012 at 1:03
  • @JackLevin: As far as generating a list of questions I'm not sure, one of our devs can answer that better. As far as whitelisting, see if you can work within the limits first -- if not lets go from there.
    – Kyle Brandt
    Jun 13, 2012 at 1:05
  • ok, I am launching my crawler, will limit to /questions/ URIs only, yfrog.com is the agent.
    – Jack Levin
    Jun 13, 2012 at 1:17
  • Looks like I hit the limit, after crawling at 1 per every three seconds, I started getting 503s. Any advice?
    – Jack Levin
    Jun 13, 2012 at 1:41
  • @JackLevin there is something wrong with your crawler: I am noticing it is crawling for /questions/388242/js/help.js and another 10 other .js files on each question it is hitting (which results in a 302 pointing back at the question), this would indicate there is some issue with you parsing the crawl tree. I would also wind back 1 per 10 seconds.
    – waffles
    Jun 13, 2012 at 2:57
  • @JackLevin also, can you please include some contact details in you user agent, see: meta.stackexchange.com/questions/130398/… for an example
    – waffles
    Jun 13, 2012 at 2:58
  • Thank you, I made some adjustments, and not hitting any .js links anymore. Here is a question "stackoverflow.com/questions/4733933/…" has a pretty good URI, if we could get a dump of those URIs from you, we an just add them into our index without doing any crawling. This would allow us to refer interested users to the stackoverflow immediately. (we have about 50 million monthly uniques).
    – Jack Levin
    Jun 13, 2012 at 4:39
  • "Yahoo bot is evil" - your doing? :D
    – Shadow Wizard Is Sad And Angry
    Feb 6, 2014 at 9:20

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .