Stack Exchange is about making the Internet better, right? They've contributed to open-source projects, spawned open-source projects, become an Open ID provider, the list goes on.

Here's one more thing that would benefit the Internet at large, providing data to Project: Honey Pot.

The idea behind Project: Honey Pot is simple:

You place pages on your site which are referenced by hidden links in all of your pages. This page (which normal users wouldn't go to, nor should they be aware of) would contain form fields (that spammers can try and submit comment spam to) as well as email addresses from donated MX records (which are used specifically to identify mail sent from harvesters).

If the form is submitted to or mail sent to those emails, then an IP record is kept at Project: Honey Pot that can then be used by people to help detect malicious content (through a simple DNS lookup).

Granted, everything is rate limited (as well as sites that aren't Google that are crawling the sites) and I'm sure there are blocks on certain IPs that have caused problems.

But there are probably tons of bots that do scrape the site, for less than honorable purposes. If so, this would allow Stack Exchange to help combat the spam problem in general with minimal cost to the sites (I've run them on two small sites for years and have even caught spammers) while at the same time, being able to benefit from the data that they and others have contributed to (if they want, through the Http:BL API).

  • 2
    Hm, how does Project Honey Pot stuff affect SEO? Crawlers still see those links
    – Ben Brocka
    Nov 13, 2012 at 20:00
  • 5
    @BenBrocka You can restrict it through robots.txt. Spam harvesters don't respect robots.txt anyways so you can keep out legitimate crawlers while letting in the bad ones (if a spam harvester is honoring the robots.txt file, they aren't very good then). Personally, I place mine in the /php subdirectory; not that my site uses PHP, but it's also doesn't scream "this is a trap". And of course, these directories are easily rotatable, dynamic, etc.
    – casperOne
    Nov 13, 2012 at 20:01
  • hm before opening I was hoping to read about Improving the Quality of Reviews: Project Honeypot but oh well let's fight spammers somewhere over there
    – gnat
    Nov 13, 2012 at 21:30
  • 3
    @gnat I'm picking the fight I think we can win =P
    – casperOne
    Nov 13, 2012 at 21:31
  • 1
    I can't be certain, but I'm reasonably sure that the majority of bots this would identify would be kept out by HAProxy anyway. Not many bots even set all required headers when making a request, and some are easily rejected by string matches (e.g. libwww-perl). I'm not so sure we'd help as much as you think we would, at least if integrated in the usual way. Still, we do have data they could probably use.
    – user50049
    Nov 14, 2012 at 0:48


You must log in to answer this question.

Browse other questions tagged .