Aside from the basic and solvable technical issues (such as, "how would you handle folks who are moderators on multiple sites?"), I feel that this feature proposal is misguided at a more fundamental level.
But my site is kinda slow
"There aren't many flags on my site" shouldn't be a reason to skip visiting the site. There is more to moderation than flag handling. If there aren't many questions, maybe the moderator could ask one. Or maybe there are some edits to be made or answers to post. I know first hand how easy it can be to stop participating in the primary purpose of the site - the Q&A - once one becomes a moderator. It's important to keep in mind that the site would still benefit from those activities in addition to handling moderation issues.
With that in mind, if the site activity is so low that moderators don't even bother showing up, the last thing we want to do is make it easier for them to stop showing up.
When a site slows down, regular users also start visiting sporadically. This means that the flags you're looking for might not even be raised. If the site's community stops participating, it ends up with non-answers, outright spam, poor questions, and generally goes into a spiral of fail. Now, of course, a single moderator or even a group of moderators can't salvage a site that's plain not working, but again - we don't need to make it easier to not show up.
Relax... have a Kit Kat
And with all that said, there's still the simple fact that we want people to take breaks and balance their time. We don't want anyone to burn out handling flags. If you are a moderator on some site that's not Sci-Fi and you just had a long, horrible day at work and you just want to zone out for a bit and read some interesting Star Wars questions on Sci-Fi, you have to be able to do so without having your own site's flag queue hanging over your head.
We appoint and elect multiple moderators and enable regular users to help out precisely to make sure the workload is shared. Nobody should ever be in a position where they have to be on the site all the time or have to watch that flag count like a hawk all the time or else problems don't get handled.
If I don't do it, who will?
What happens sometimes is that the teams are unbalanced. A small subset of the moderation team is greatly more active than the rest, and so all the moderators who don't get to do much get out of habit of checking in on issues. Sometimes folks get elected and then never do much with the diamond. All of this happens. The solution is to balance the team better, not to put the responsibility for the workload on that small subset of the team.
As a side note, that's basically what's going to happen on Drupal. We're going to take a closer look at the team dynamic there and see what (if anything) we need to do to balance things out.
This holds for every site out there: if you are having issues with moderation (too much, not enough, the wrong kind) or site activity, come talk to us. And if your moderators are doing a great job, be sure to let them know. It can be a bit thankless sometimes. :)