If an attempt at humor within an answer is otherwise not valuable, and if the question is otherwise serious, and if the answer is distracting, then I think downvoting is justified.
These sorts of "humorous" answers are akin to some jack-ass making jokes or throwing pies at someone trying to change a tire. Humor is fine as a complement to getting things done, when it gets in the way of work, then it becomes a problem. Because, to be honest, a lot of this "humor" isn't as funny as the posters think it is.
As someone above pointed out, valuing humor as highly as concrete help is a recipe for a rapidly diminishing signal to noise ratio, as seen on slashdot and elsewhere.
If you want to make a joke in a serious topic, put it in a comment.
If the entire Q&A thread is humorous, then downvoting isn't justified (unless the joke isn't funny, live by the sword, etc.)
Edit: keep in mind this is coming from someone who posted a funny answer to the (now deleted) "How to make a peanut butter and jelly sandwich" question. Humor is good, but you have to be careful from letting it get out of control.