A spam vote (like an offensive vote) is very simple; there is no retraction (or even obvious visibility). But don't panic if you do it by accident; it needs about 6 such votes before the system will automatically step in.
Before that, items with spam/offensive votes are visible to users with 10k rep, and to ♦ moderators. The ♦ moderators will check the list (which is usually empty or short) periodically and investigate. If the item is obvious spam, it'll usually be deleted. If it is a false positive (i.e. innocent), we'll clear the flag. If it is unclear, we might let it ride for a while to see if any more community members agree it is spam.
Flagging for a moderator is different; you need to give a reason, and only the ♦ moderators can see it. This is made really clear to us (a red counter at the top of every page), and we'll investigate ASAP. Assuming it is a reasonable complaint, of course ;-p