I always did the same thing, and I (so far) have been up/down voting far more. Instead of keying in on a specific score I feel appropriate for that comment, I'm keying in on just the three options: Up, Down, No change. Without having a visible score, I am no longer constrained by "If I up/down vote this, it will take it out of the range I want it to be in". Instead I'm just up/down voting as I want to.