My website follows robots.txt, and only has at most 1 connection to a host at one time. It never fetches a page more than once an hour. Well within reason (I hope), so I'm wondering why I'm banned and how to resolve it?

Many users of my service have asked to graph their SO numbers.

Cralwer's IP:

Cralwer's DNS: webnumbr.com

And example of me trying to fetch an SO page (30 second timeout) http://webnumbr.com/create?url=https://stackoverflow.com/users/90025/paul-tarjan

  • 9
    That's a cool website, by the way. – deleted Feb 7 '10 at 5:51
  • 1
    I agree. Neat idea. – John Gietzen Feb 7 '10 at 6:18
  • 3
    Thanks! :) I started it to save money on flights, but have found tons of uses already. Hopefully it helps you too. – Paul Tarjan Feb 7 '10 at 7:11

that IP is not in our ban list.

There is a similar IP in the ban list for 2 GB of bandwidth used in a day:

-A INPUT -s -j DROP  
  • 1
    Wow, nice way to find the IP Koper. Clever. I guess outbound traffic is on another interface than inbound... weird... I'll work with you offline about the ban (not really sure why...) – Paul Tarjan Feb 7 '10 at 8:32
  • @Paul, Koper's trick also shows some funny behaviour while pointing out the number: I can only select specific digits of the IP address? (Safari and Firefox on a Mac.) – Arjan Feb 7 '10 at 22:01
  • @Arjan, view the source of that page. It looks like that site really doesn't want you scraping the number and puts all sorts of markup and comments to make that suck. You can always click something, and then in advanced write an XPath that will select exactly what you want. – Paul Tarjan Feb 9 '10 at 8:34
  • 1
    @Paul, indeed, nice. :-) They have even put a remark in that HTML source, which I assume you've seen: <!--Please set your code to scrape your IP from www.whatismyip.com/automation/n09230945.asp For more info, please see our "Recommended Automation Practices" thread in the Forum.--> – Arjan Feb 9 '10 at 9:15

Send a mail explaining this to team@stackoverflow.com. All non Google/Bing/Yahoo crawlers are being filtered both by User Agent and IP address for security/performance reasons.

Jeff & Co. are the ones you have to talk to in order to be unfiltered for crawling.

  • @Koper: I understand what you are saying, but this is not any website. This is a big target for illicit crawlers because of X reasons. – perbert Feb 7 '10 at 6:52
  • Emailed, thank you. I won't get into the argument :) – Paul Tarjan Feb 7 '10 at 7:26
  • (Deleted my rant since per Jeff's answer this is a blacklist and not a whitelist) – Thomas Bonini Feb 7 '10 at 8:14

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .