11

My website follows robots.txt, and only has at most one connection to a host at one time. It never fetches a page more than once an hour. Well within reason (I hope), so I'm wondering why I'm banned and how to resolve it?

Many users of my service have asked to graph their Stack Overflow numbers.

The crawler's IP address: 208.83.223.136

The crawler's DNS: webnumbr.com

And an example of me trying to fetch an Stack Overflow page (30-second timeout) http://webnumbr.com/create?url=https://stackoverflow.com/users/90025/paul-tarjan

Improve this question
3
  • 9
    That's a cool website, by the way.
    – deleted
    Feb 7, 2010 at 5:51
  • 1
    I agree. Neat idea.
    – John Gietzen
    Feb 7, 2010 at 6:18
  • 3
    Thanks! :) I started it to save money on flights, but have found tons of uses already. Hopefully it helps you too.
    – Paul Tarjan
    Feb 7, 2010 at 7:11

2 Answers 2

Reset to default
5

That IP address is not in our ban list.

There is a similar IP address in the ban list for 2 GB of bandwidth used in a day:

-A INPUT -s 208.83.223.139/32 -j DROP
Improve this answer
5
  • That is his. webnumbr.com/create?url=http://www.whatismyip.com
    – Thomas Bonini
    Feb 7, 2010 at 8:13
  • 1
    Wow, nice way to find the IP Koper. Clever. I guess outbound traffic is on another interface than inbound... weird... I'll work with you offline about the ban (not really sure why...)
    – Paul Tarjan
    Feb 7, 2010 at 8:32
  • @Paul, Koper's trick also shows some funny behaviour while pointing out the number: I can only select specific digits of the IP address? (Safari and Firefox on a Mac.)
    – Arjan
    Feb 7, 2010 at 22:01
  • @Arjan, view the source of that page. It looks like that site really doesn't want you scraping the number and puts all sorts of markup and comments to make that suck. You can always click something, and then in advanced write an XPath that will select exactly what you want.
    – Paul Tarjan
    Feb 9, 2010 at 8:34
  • 1
    @Paul, indeed, nice. :-) They have even put a remark in that HTML source, which I assume you've seen: <!--Please set your code to scrape your IP from www.whatismyip.com/automation/n09230945.asp For more info, please see our "Recommended Automation Practices" thread in the Forum.-->
    – Arjan
    Feb 9, 2010 at 9:15
4

Send a mail explaining this to team@stackoverflow.com. All non Google/Bing/Yahoo crawlers are being filtered both by User Agent and IP address for security/performance reasons.

Jeff & Co. are the ones you have to talk to in order to be unfiltered for crawling.

Improve this answer
2
  • @Koper: I understand what you are saying, but this is not any website. This is a big target for illicit crawlers because of X reasons.
    – perbert
    Feb 7, 2010 at 6:52
  • Emailed, thank you. I won't get into the argument :)
    – Paul Tarjan
    Feb 7, 2010 at 7:26

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .