Occasionally, I downvote a bad answer but the -1 is not subtracted immediately and may sometimes take a few hours. Why is that the case? Why is it not immediate like when gaining a reputation for an upvoted post/comment?
As Catija points out there's no notification of reputation loss, the next positive notification will be reduced by the amount lost.
However, the -1 for down-voting and the -2 for being down-voted are delayed and delayed by different amounts. This is done to help preserve the anonymity of voting.
Imagine the scenario. Someone receives a down-vote and notes the time at which it occurred. They suspect that a specific user has been down-voting them so they go to that user's profile. If the rep loss was immediate they'd see the correlation between their rep loss and their supposed nemesis's rep loss and possibly start a vendetta.
With the delay there's no way that anyone can know for sure that the rep loss was tied to a specific down-vote. That way the opportunity for retaliation is reduced or even eliminated.
It's not perfect, as it's security through obscurity, but it does go a long way.