That's tricky. Let's say we upvote "valuable" comments not just merely those that we agree with. The problem is, that we will see comments fitting with our own views as valuable. The same problem is with comments we don't agree with. So honestly, upvotes are just as much about agreement as they are about value in many cases. Because of that I think downvoting comment simply as the sign of disagreement is OK, if even for the sake of symmetry. You don't expect your car to turn to the left when you turn the steering wheel to the left, but slow down, when you turn it to the right. If you click up-arrow to express agreement, you may click down-arrow to express disagreement. I do sometimes upvote comments even if I don't agree with them, but I cannot remember downvoting a comment I do agree with.