I think we should leave them be, because the downvotes make it clear that it is wrong, and showing that something is wrong is useful information.

For example, if I read an answer that says "electrons are made of chocolate", and it has -17 downvotes, I will assume that electrons are *not* made of chocolate, which is correct. I actually learned something.

If the answer was deleted, then maybe the idea of chocolate electrons could creep into my mind some day.

It's like a vaccine against wrong information!