What is meta? ×
Meta Stack Exchange is where users like you discuss bugs, features, and support issues that affect the software powering all 128 Stack Exchange communities.

Hi guys, I've made a Python script which you can use to get notifications of the latest questions. It scrapes the unanswered-questions page and informs you via libnotify. Just click on the notification button to open a browser window on the question.

  • Why not use an RSS reader? Well, this is more customized than an RSS reader. Not only does it tell you that there are new messages, but it tells you their title and tags without you having to switch windows. Moreover, it identifies questions by their IDs rather than by their titles, so if a question changes its title, you won't get a new notification.
  • Why not read from the RSS feed? It's generally updated less quickly. In the highly competitive world of SO, a couple of minutes can make a difference in rep.
  • How do I use it? Copy and paste into your favorite editor, edit the settings at the top, save and run it in the background. You may not notice anything for a few minutes, since it doesn't show anything until it encounters a new question.
  • Dependencies? You should probably get this from your package manager, but I've included links to the sources as well: pygobject, python-notify, python-lxml (with installation instructions).
  • Isn't this not-a-question? Yeah, but it seems that past users have submitted their SO 'addons' here as well.

Screenshot:

screenshot

Do let me know if you find it useful (or useless), and if there are any bugs, or if you think there should be more features.

Minor updates:

  • The last script gave the same timeout whether one or three notifications showed up at once. I have changed it to increment the timeout per additional notification; the topmost notification expires first, so read them from top to bottom.
  • The script now sends out a notification after initialization, to let you know that something's alive.
#! /usr/bin/python

import subprocess
import time
import gobject
import pynotify
import urllib
from lxml import etree
import thread

# CONFIGURATION

browser = "chrome" # however you call it at the command line
notify_timeout = 5000 # ms
tagnames = "c++ or c or python or optimization or .net or asp.net or c# or java"
scrape_site = "http://stackoverflow.com/" # or serverfault, or superuser...
scrape_url = scrape_site + "unanswered/tagged?tagnames=" + urllib.quote(tagnames) + "&tab=newest"
refresh_rate = 60 # seconds

# END CONFIGURATION

def find_new_qns(url):
    old_ids = set()
    while True:
        try:
            data = urllib.urlopen(url).read(1000000)
        except Exception as inst: # IOErrors happen now and then, just ignore them
            # print type(inst), inst
            continue

        html = etree.HTML(data)
        summaries = html.xpath("//div[@class='question-summary']")
        if len(old_ids) == 0:
            old_ids = set([summary.attrib["id"] for summary in summaries])

        new_qn_count = 0
        for summary in summaries:
            if summary.attrib["id"] not in old_ids:
                new_qn_count += 1
                old_ids.add(summary.attrib["id"])
                title = summary.xpath(
                    "descendant::a[@class='question-hyperlink']")[0]
                tags = \
                    summary.xpath("descendant::a[@class='post-tag']/text()")
                notify(title.text, " ".join(tags), scrape_site + title.attrib["href"], notify_timeout*new_qn_count**0.8)

        if len(old_ids) > 1000:
            old_ids = set([summary.attrib["id"] for summary in summaries])
        time.sleep(refresh_rate)

def notify(title, message, target_url, timeout):
    def notify_thread(title, message, target_url):
        note = pynotify.Notification(title, message)
        if target_url is not None:
            note.add_action(target_url, "View", lambda n,url: subprocess.call([browser, url]))
        note.connect("closed", lambda n: loop.quit())
        note.set_timeout(int(timeout))
        note.show()
        loop = gobject.MainLoop()
        loop.run()
    thread.start_new_thread(notify_thread, (title, message, target_url))

if __name__ == '__main__':
    gobject.threads_init()
    pynotify.init("stackoverflow-notify")
    # if we've got this far, the libraries are basically there.
    # ... hopefully the versions are sufficiently recent.
    # let the user know we're alive
    notify("Stack Exchange Notifier Initialized.", "", None, notify_timeout)
    find_new_qns(scrape_url)
share|improve this question
    
can you post the links/directions for the additional libraries? –  OscarRyz Nov 20 '09 at 13:52
    
I've added the sources, and changed the names to what they'd typically be called in a package manager.. I'm using pacman, but you'd probably be able to apt-get/yum/etc the stuff. I believe they're pretty popular libraries. –  splicer Nov 20 '09 at 17:56
3  
Does this script fetch the html uncompressed? If so, can you modify it to fetch gzipped, please? –  Jarrod Dixon Feb 25 '10 at 3:16
3  
Also, if you use this script, please append your user id to the user agent, e.g. Python-urllib/1.17+(user/3/jarrod-dixon) - this helps us know not to ban you for excessive scraping! –  Jarrod Dixon Feb 25 '10 at 3:18
    
Another suggestion to keep the payload size smaller... add a &pagesize=5 to the end of the query string to only get the newest 5 rather than 30 or whatever the default is (likelihood of getting more than 5 new questions in a limited set of tags in 60 seconds is probably fairly slim). –  Alconja Feb 25 '10 at 4:45
add comment

3 Answers 3

why don't you just use the RSS feed and write a script to filter that?

share|improve this answer
add comment

Be aware that I got IP-banned from SO for using the script with default 60-second polling. According to this thread - Etiquette of Screen-scraping Stack Overflow? - scraping more frequently than 15 minutes is not allowed. So far the SO team has ignored requests to lift the IP block.

share|improve this answer
add comment

Is this safe? I mean, is Jeff going to ban me for consuming too much bandwidth? Shouldn't you use the publicly exposed methods of getting this info instead of consuming precious internet/server resources to scrape the whole page?

share|improve this answer
    
welll, I'd like to find out as well.. perhaps Jeff will enlighten us in the near future. I was actually working on a RSS version of this before I realized that posts there were only getting updated in batches. –  splicer Nov 20 '09 at 13:25
8  
well, I just did a comparison, and the page-scrape actually consumes slightly less bandwith. The RSS feed actually gives you the full body of the post, while the main page gives you excerpts; I think that accounts for the difference. The only issue now is whether this polling is too frequent. –  splicer Nov 20 '09 at 13:32
    
+1 for testing. –  snicker Nov 20 '09 at 16:50
    
.. but not for the script itself? :P –  splicer Nov 20 '09 at 18:08
add comment

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .