It's not possible to identify who cast a downvote.1
The best a person can do is to make an educated guess that a person who left a critical comment around the same time also cast the downvote, or something like that. In the absence of more information, I suspect that's probably what happened here. But those guesses are not always accurate, and the person who guesses has no way of knowing whether their guess is correct unless you confirm it to them by responding.
Speculating or making assumptions about who cast a downvote is officially discouraged. If someone accuses you of having downvoted in a comment, whether or not they're right, there's a pretty good chance that comment qualifies as either "unfriendly or unkind" or "no longer needed", so please feel free to cast any applicable flag. Meanwhile, I would suggest simply ignoring the accusation.
1 To be more precise: nobody except a few high-level Stack Exchange employees has access to the information about who cast an individual (up- or) downvote. Not even diamond moderators. For completeness, I should note that diamond mods do have access to some information about voting patterns; for instance, if one account is giving an unusually large number of votes to another account, our vote information tools can help detect that, but they don't show the specific posts on which those votes were cast.