I think that the reputation you start off with when linking your account with other Stack Exchange sites should be increased. For example, I was recently linked to Print 1 to 1000 in C++ without semi-colons on Code Golf, and one of the highest upvoted answers isn't even legal C++, but I can't downvote it- requires 125 rep. In fact, most of those answers aren't legal C++.
Quick edit: Illegal C++ is not the same as exhibiting Undefined Behaviour. Illegal C++ will not compile. Undefined Behaviour will do something undefined at run-time, such as signals or structured exceptions or debugger breaking or pretty much anything. There is a clear and distinct difference and accepting UB does not imply accepting code that doesn't compile.
This is the same with other privileges too, not just downvoting. For example, on SO, I can edit posts without needing to be reviewed, which is fine and great. Ever had a complaint about an edit I've made? Not to my knowledge. But if I want to edit a post on Game Development, then I still have to excuse myself and ask other people to approve it- even though I'm perfectly trusted to edit other people's posts here. Does the Stack Exchange network think that I can be trusted to edit other people's posts without approval or not? The SE Network is one network. I'm one person. There is one, fixed, relationship here.
The fundamental trouble here is that some privileges are about trust, such as editing without needing approval, and reputation is about knowing the subject matter and having people agree with you and being around for a while without getting banned for horrific crimes against humanity. For a person that's new to Stack Exchange then I can totally understand the need to link them, but since I'm not new to Stack Exchange, then it would be nice if you could infer from the fact that I can be trusted not to abuse my edit, downvote, etc privileges here that I might not abuse them on the other sites, too. Alternatively, the reputation on new linked accounts could be proportional to my current rep on SO, instead of a fixed value.
Edit: You're completely missing the point. When I'm on SO, do you think I go editing all the Bash questions because I've got high rep in the C++ tag? Of course not. I only edit the questions that I'm qualified to edit- and that won't be different just because I'm not qualified to edit questions about Photography instead of not qualified to edit questions about Bash. On the other hand, there will be questions on other sites that cover material that I will be qualified to edit.