I just tried to down vote an answer. The vote count went up. I hit the down button again, and the count went up again! I'm afraid to try it again. FYI
Who knows, maybe two other people upvoted the answer at the same time? I wouldn't worry too much about the number. Just refresh the page or open it in a new tab and see if that shows a more reasonable number. The SE database stores individual votes, and the post scores are derived from that, so if the number is wrong, it will work itself out next time the post score is recalculated.
What's more important is to check that the vote you cast on the answer (if any) is the one you intended to cast. Refresh or reopen the page and see whether one of the arrows is highlighted, indicating the vote that has been recorded for you. If that's not the vote that you wanted to cast, click the arrow again to undo it. But you only have 5 minutes to do this, so in this case it's too late, unless the answer is edited.