tl;dr: yes, it's doable, and could offer interesting new ways to solve the question of displaying question-relevant reputation, and of filtering displayed content to suit each user's interests.
You've got a good idea. It's just so different to how things happen here, and too similar to other sites that have failed to establish quality content, and folk won't like it for those reasons.
But there's more to it. Let's take a look
How do we separate content? And why in that form?
The present distinction between separating-content-by-site and separating-content-by-tag is somewhat arbitrary (it's a bit of an art, a bit of a science, and a bit of evolutionary legacy, aka lavaflow - it just set where it is, and we work around it). It works in some cases: some sites are the right size to be a site in their own right, and where expertise across the whole topic means something. In those cases, sharing a questions page and rep across that whole topic makes sense.
However, Stack Overflow is much much larger than that. If a Ruby expert answers a Haskell question, all you see is their SO rep, earnt from their Ruby answers. So, if showing other people's rep has any meaning at all, that meaning is probably being undermined in that case.
Physics.SE is smaller, but has similar problems: three of its very highest rep users each believe in some extremely non-mainstream physics in some particular areas, as well as being very strong in other physics areas.
The over-arching Stack Exchange site tries to make up for the scattering of connected information across different sites, but doesn't yet do it very well.
Two problems; each has more than one solution (TMTOWTDI)
So, there are two problems here: displayed rep that appears to be topically relevant, but which may well not be. And connected content that's scattered across sites. The arrangement in place at the moment is an evolving pragmatic compromise between topic-specific self-contained sites, and a multi-topic expert Q&A bank. But the arrangement we have at the moment, is by no means the only way to do it. The distribution of content looks radically different to how it looked four years ago, and there's every chance that it could look radically different in four years time.
I think your suggestion has merit. It needs work, to thrash out what rep would be displayed for a particular poster on any specific question or answer, and how people filtered the content they wanted to see. But it's not intrinsically unworkable, and could offer routes to more flexible and interesting ways to resolve these difficult issues of displaying a reputation score that's relevant to the question; and of filtering the content that's displayed to users to best suit their interests.
(all this is predicated on StackExchange continuing to display a user's rep alongside the content they contribute. That may or may not be the case in the future. And it's worth thinking about why it's displayed, and what is gained by it.)