A friend and I were discussing this and came up with the following solution: have an "I don't know" button, for when you see an answer, you realize that it is a legitimate question but simply don't know the answer. Then people could subscribe to the questions that have at least 50 "I don't know"s or something. The solution has some issues (such as, there are different areas, not just levels of expertise, but maybe the existing tag system takes care of that?), but it's something.
I'm wondering if people have thought about this and have other ideas to solve this problem. Maybe a solution is already in place and I simply don't know about it?