Others have answeredanswers regarding downvotes, but here is my suggestion.
Consider giving explanations when downvoting questions and/or answers. Knowing the problem with their contribution will help them rectify the problem and/or make better contributions in the future.
Of course this does not necessitate commenting, but leaves it to the discretion of the possible downvoter.