Why are lengthy discussions now allowed in comments?
I hope Manishearth won't mind if I steal his comment, but they're not. The reason it seems like they might be is that we usually don't get around to cleaning them up unless someone flags them.
To be more precise: Stack Exchange gives us (moderators) some slightly contradictory guidance on comments. On one hand, the intent of comments on main sites is that they are supposed to be transient, and used for suggestions, corrections and clarifications, not for discussion. However, we're also advised not to indiscriminately delete comments unless they've degenerated into an argument. As you can tell, these two guidelines aren't quite compatible with each other. Different moderators and different sites have chosen different approaches to dealing with comments, within this spectrum, and our compromise here is to remove comments when their usefulness has passed and they are brought to our attention, but otherwise not to look too hard for comment discussions to delete. We rely on the community and the system to find the cases where comments really need to be cleaned up.