I upvoted a post mistakenly. Then I tried to cancel my vote but I couldn't because my reputation isn't enough. How can I cancel my vote?
You cancel a vote by re-clicking the arrow for that vote, which you have the rep for that vote. That is, you cancel a upvote by clicking the up-arrow again. And the rep limit for that is the same as for casting the vote in the first place.
It sounds like you tried to cast a downvote which is a separate action.
That said, there is a time limit for canceling votes. After 5 minutes (I think) you won't be able to cancel that vote unless the post has been edited in the mean time.