If I downvote an answer there is a -1 reputation change. Even though, downvotes and upvotes are supposed to be anonymous. There is a slight chance that user might visit my profile and can see my -1 reputation change from my profile.
PS: I couldn't find any answer that I could downvote, but hopefully you all get my point.