I have many accounts on SE sites with little or no activity, and I get invitations to vote in their elections despite of that. Sure, sensible users will not vote, but sometimes they will, just out of curiosity or because they don't know how much their vote matters. And many people like that voting on superficial indicators of candidate quality such as candidate rating and the length of their nomination posts, skews the results.
Not displaying the invitation to visit the voting page is a bad idea for various reasons, so I'm not proposing that.
What I propose is for the users who have no recent activity to add a notice at the top of the election page saying roughly this:
It seems like you don't have any recent activity on this site. If you don't have enough experience using this particular site, please don't cast yourconsider refraining from voting or researching the candidates to make an educated vote.
I would add something about knowing mods from other sites, and if they were good there, it's ok to vote for them, but that might not be a good idea because they might not be as good as other candidates on a SE site with a different topic.
If you think users should be encouraged to vote regardless of their inactivity on a site, I'd like to hear why.