I just downvoted an answer. (In this case because it was more of a comment than an answer.) As soon as I downvoted, I got a -1 in my reputation. Is it supposed to be like this, or is it some technical fault?
Yes, there is a 1 rep point hit for downvoting answers. There isn't any reputation hit for downvoting questions.
I've noticed that really bad answers tend to attract 3-4 downvotes and then the author is motivated to remove the answer. When an answer that you downvoted is removed you get your reputation back.
The reputation loss does make you think twice about downvoting an answer but don't let it discourage you. Wrong or poor answers need to be downvoted.