Do I have to award the bounty even if the answer does not deserve it and is downvoted? This is in relation to this question of mine where there is only one and that too downvoted answer..
You never have to award a bounty.
If you choose not to manually award a bounty, it may be automatically awarded to an answer, but not if that answer is downvoted. From the help center:
If you do not award your bounty within 7 days (plus the grace period), the highest voted answer created after the bounty started with a minimum score of 2 will be awarded half the bounty amount. If two or more eligible answers have the same score (i.e., their scores are tied), the oldest answer is awarded the bounty. If there's no answer meeting those criteria, the bounty is not awarded to anyone.
In your specific case, the bounty will not be automatically awarded.
Do I have to award the bounty even if the answer does not deserve it and is downvoted?
No you don't, but there is a certain "honour" at work here. I've offered a bounty before, and awarded it even though I wasn't happy with the answers. See for example this question. As a result people know that they can trust me. If you offer a bounty and then don't award it, next time you won't get one answer, you'll get none.
This is in relation to this question of mine where there is only one and that too downvoted answer.
You should ask the posters here if they downvoted that answer. And if so, why they didn't offer an answer of their own. Do note that there are some downvoting issues here on physics Stack Exchange, see this meta question. Always remember that physics is not a democracy. Votes don't make a right answer wrong, or vice versa. You should look to the evidence and explanation and references, and ask for elaboration in the comments. Or ask a new question to tease out the issue and get to the underlying problem.