Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 158 Stack Exchange communities.

What is meta?
Here's how it works:
  1. Any Stack Exchange user can ask a question
  2. The community provides support, votes on ideas, and reports bugs
  3. Your voice helps shape the way Stack Exchange operates

My website follows robots.txt, and only has at most 1 connection to a host at one time. It never fetches a page more than once an hour. Well within reason (I hope), so I'm wondering why I'm banned and how to resolve it?

Many users of my service have asked to graph their SO numbers.

Cralwer's IP:

Cralwer's DNS:

And example of me trying to fetch an SO page (30 second timeout)

share|improve this question
That's a cool website, by the way. – deleted Feb 7 '10 at 5:51
I agree. Neat idea. – John Gietzen Feb 7 '10 at 6:18
Thanks! :) I started it to save money on flights, but have found tons of uses already. Hopefully it helps you too. – Paul Tarjan Feb 7 '10 at 7:11

that IP is not in our ban list.

There is a similar IP in the ban list for 2 GB of bandwidth used in a day:

-A INPUT -s -j DROP  
share|improve this answer
Wow, nice way to find the IP Koper. Clever. I guess outbound traffic is on another interface than inbound... weird... I'll work with you offline about the ban (not really sure why...) – Paul Tarjan Feb 7 '10 at 8:32
@Paul, Koper's trick also shows some funny behaviour while pointing out the number: I can only select specific digits of the IP address? (Safari and Firefox on a Mac.) – Arjan Feb 7 '10 at 22:01
@Arjan, view the source of that page. It looks like that site really doesn't want you scraping the number and puts all sorts of markup and comments to make that suck. You can always click something, and then in advanced write an XPath that will select exactly what you want. – Paul Tarjan Feb 9 '10 at 8:34
@Paul, indeed, nice. :-) They have even put a remark in that HTML source, which I assume you've seen: <!--Please set your code to scrape your IP from For more info, please see our "Recommended Automation Practices" thread in the Forum.--> – Arjan Feb 9 '10 at 9:15

Send a mail explaining this to All non Google/Bing/Yahoo crawlers are being filtered both by User Agent and IP address for security/performance reasons.

Jeff & Co. are the ones you have to talk to in order to be unfiltered for crawling.

share|improve this answer
@Koper: I understand what you are saying, but this is not any website. This is a big target for illicit crawlers because of X reasons. – perbert Feb 7 '10 at 6:52
Emailed, thank you. I won't get into the argument :) – Paul Tarjan Feb 7 '10 at 7:26
(Deleted my rant since per Jeff's answer this is a blacklist and not a whitelist) – Andreas Bonini Feb 7 '10 at 8:14

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .