There could be an arrow pointing down, the same as in this image:
It could open a window just like the Stack Exchange icon and have some tabs, like:
And a link somewhere that would lead to
/users/recent/, where you can see everything.
Perhaps the envelope icon could change to fit. The icon would continue redirecting to
/users/recent/ and the down arrow would open that nice window. The inbox would display only changes from the current site and indicate how many new things are out there. Just like the global inbox.
Well, it's just a thought. I think changing it "completely" would automatically fix the bugs and give a nice feature.