If you down voted an answer then you lose 1 reputation point, which is only viewable to you. Since I can't see this on your own reputation page I'm guessing that's what happened. You will probably be able to see it on that page though (and any other reputation change history of yours).
You can't lose/gain reputation for comments in any way.