Hi guys, I've made a Python script which you can use to get notifications of the latest questions. It scrapes the unanswered-questions page and informs you via libnotify. Just click on the notification button to open a browser window on the question.

  • Why not use an RSS reader? Well, this is more customized than an RSS reader. Not only does it tell you that there are new messages, but it tells you their title and tags without you having to switch windows. Moreover, it identifies questions by their IDs rather than by their titles, so if a question changes its title, you won't get a new notification.
  • Why not read from the RSS feed? It's generally updated less quickly. In the highly competitive world of SO, a couple of minutes can make a difference in rep.
  • How do I use it? Copy and paste into your favorite editor, edit the settings at the top, save and run it in the background. You may not notice anything for a few minutes, since it doesn't show anything until it encounters a new question.
  • Dependencies? You should probably get this from your package manager, but I've included links to the sources as well: pygobject, python-notify, python-lxml (with installation instructions).
  • Isn't this not-a-question? Yeah, but it seems that past users have submitted their SO 'addons' here as well.


screenshot http://inlinethumb10.webshots.com/43145/2370085690102701653S600x600Q85.jpg

Do let me know if you find it useful (or useless), and if there are any bugs, or if you think there should be more features.

Minor updates:

  • The last script gave the same timeout whether one or three notifications showed up at once. I have changed it to increment the timeout per additional notification; the topmost notification expires first, so read them from top to bottom.
  • The script now sends out a notification after initialization, to let you know that something's alive.
#! /usr/bin/python

import subprocess
import time
import gobject
import pynotify
import urllib
from lxml import etree
import thread


browser = "chrome" # however you call it at the command line
notify_timeout = 5000 # ms
tagnames = "c++ or c or python or optimization or .net or asp.net or c# or java"
scrape_site = "http://stackoverflow.com/" # or serverfault, or superuser...
scrape_url = scrape_site + "unanswered/tagged?tagnames=" + urllib.quote(tagnames) + "&tab=newest"
refresh_rate = 60 # seconds


def find_new_qns(url):
    old_ids = set()
    while True:
            data = urllib.urlopen(url).read(1000000)
        except Exception as inst: # IOErrors happen now and then, just ignore them
            # print type(inst), inst

        html = etree.HTML(data)
        summaries = html.xpath("//div[@class='question-summary']")
        if len(old_ids) == 0:
            old_ids = set([summary.attrib["id"] for summary in summaries])

        new_qn_count = 0
        for summary in summaries:
            if summary.attrib["id"] not in old_ids:
                new_qn_count += 1
                title = summary.xpath(
                tags = \
                notify(title.text, " ".join(tags), scrape_site + title.attrib["href"], notify_timeout*new_qn_count**0.8)

        if len(old_ids) > 1000:
            old_ids = set([summary.attrib["id"] for summary in summaries])

def notify(title, message, target_url, timeout):
    def notify_thread(title, message, target_url):
        note = pynotify.Notification(title, message)
        if target_url is not None:
            note.add_action(target_url, "View", lambda n,url: subprocess.call([browser, url]))
        note.connect("closed", lambda n: loop.quit())
        loop = gobject.MainLoop()
    thread.start_new_thread(notify_thread, (title, message, target_url))

if __name__ == '__main__':
    # if we've got this far, the libraries are basically there.
    # ... hopefully the versions are sufficiently recent.
    # let the user know we're alive
    notify("Stack Exchange Notifier Initialized.", "", None, notify_timeout)
  • can you post the links/directions for the additional libraries? – OscarRyz Nov 20 '09 at 13:52
  • I've added the sources, and changed the names to what they'd typically be called in a package manager.. I'm using pacman, but you'd probably be able to apt-get/yum/etc the stuff. I believe they're pretty popular libraries. – splicer Nov 20 '09 at 17:56
  • 3
    Does this script fetch the html uncompressed? If so, can you modify it to fetch gzipped, please? – Jarrod Dixon Feb 25 '10 at 3:16
  • 3
    Also, if you use this script, please append your user id to the user agent, e.g. Python-urllib/1.17+(user/3/jarrod-dixon) - this helps us know not to ban you for excessive scraping! – Jarrod Dixon Feb 25 '10 at 3:18
  • Another suggestion to keep the payload size smaller... add a &pagesize=5 to the end of the query string to only get the newest 5 rather than 30 or whatever the default is (likelihood of getting more than 5 new questions in a limited set of tags in 60 seconds is probably fairly slim). – Alconja Feb 25 '10 at 4:45
  • 2
    I encountered this question in the Close Vote review queue; it's probably from before the existence of the API and should be rewritten and posted on Stack Apps. – Glorfindel Apr 29 at 19:28

Is this safe? I mean, is Jeff going to ban me for consuming too much bandwidth? Shouldn't you use the publicly exposed methods of getting this info instead of consuming precious internet/server resources to scrape the whole page?

| improve this answer | |
  • welll, I'd like to find out as well.. perhaps Jeff will enlighten us in the near future. I was actually working on a RSS version of this before I realized that posts there were only getting updated in batches. – splicer Nov 20 '09 at 13:25
  • 9
    well, I just did a comparison, and the page-scrape actually consumes slightly less bandwith. The RSS feed actually gives you the full body of the post, while the main page gives you excerpts; I think that accounts for the difference. The only issue now is whether this polling is too frequent. – splicer Nov 20 '09 at 13:32
  • .. but not for the script itself? :P – splicer Nov 20 '09 at 18:08

Be aware that I got IP-banned from SO for using the script with default 60-second polling. According to this thread - Etiquette of Screen-scraping Stack Overflow? - scraping more frequently than 15 minutes is not allowed. So far the SO team has ignored requests to lift the IP block.


why don't you just use the RSS feed and write a script to filter that?

| improve this answer | |

Not the answer you're looking for? Browse other questions tagged .