If you down voted an answer then you lose 1 reputation point, which is only viewable to you. Since I can't see this on your own [reputation page](https://physics.stackexchange.com/users/159486/semoi?tab=reputation) I'm guessing that's what happened. You will probably be able to see it on that page though (and any other reputation change history of yours).

You can't lose/gain reputation for comments in any way.