Today I downvoted an answer and flagged it, too. Now I get negative scores. Presumably, because I wrote a comment on a bad answer, which is downvoted by others too. Although, I do not care too much about my score, I am interested in understanding the motivation for the negative scores. Is it a bug?
If you down voted an answer then you lose 1 reputation point, which is only viewable to you. Since I can't see this on your own reputation page I'm guessing that's what happened. You will probably be able to see it on that page though (and any other reputation change history of yours).
You can't lose/gain reputation for comments in any way.