Note that each site lists privileges that can be earned. Find it on each specific site's help section, under the heading privileges. On Meta. SE, e.g., any user who has earned 100 or more in reputation, can downvote posts (and those downvotes are accompanied by -2 in rep to the poster).
On math.se, on the other hand, and most SE sites, a user needs 125+ rep before they can downvote a post in such a way.
Any/all downvotes cast by a user who has not yet met that rep threshold will not be seen by other users, nor will there be any point loss to the poster. Those early votes will have no future effect on the poster receiving an unqualifying downvote from a user that lacks the privilege to downvote at the time of their downvote, even after the downvoter later gains more rep.