I tend to mark things as "favorite" more as a bookmark than anything else.
However, sometimes I go back through them and realize that there are those I just don't care about anymore and I hate clutter. Obviously unmarking them as a favorite is the answer. Then again, I feel bad about "unfavoriting" something because I know the OP is going to lose the points I had originally given.
How about if someone unfavorites an item, instead of taking those points back, just let them have it. After it was a favorite for awhile.. just not forever.