Is it wise to subtract your reputation when the question is deleted? I personally think that is a bad idea because somebody when answering the question, regardless how non-relevant question it was, will invest some time to answer it. So, while I think posts that do not meet SO criteria should be deleted, the reputation and score associated with that question should be kept. Why do SO want to take away from the people who took time to answer the question after giving them points?
There is already a special case in place for this: If your post has a score >= 3 and was visible for more than 2 months before it was deleted, you keep all reputation gained from it.
[...] somebody when answering the question, regardless how non-relevant question it was, will invest some time to answer it.
Right, and there's the problem. If the question is so blatantly off-topic or of very low quality that it will be deleted anyway, then you shouldn't waste your time answering it. Encouraging people who do that is a bad idea. It sends the wrong signal ("we don't accept these questions, but we're gonna answer 'em anyway"), and prevents them from being pruned automatically.