If you down voted an answer then you lose 1 reputation point, which is only viewable to you. Since I can't see this on your own reputation page I'm guessing that's what happened. You will probably be able to see it on that page though (and any other reputation change history of yours).
You can't lose/gain reputation for comments in any way.
The number on the reputation tab on your profile is the amount of reputation you've gained since the last time you personally clicked on that tab. That fits the pattern with numbers on all of the other tabs.