As the title suggests, I would like to hear everyone's opinion on who should be allowed to downvote. I have seen youngsters downvote good answers simply because they could not understand it. They should instead request a clearer explanation rather than downvoting. What do you think ? Should the reputation for downvote be increased?
Let's examine the facts: according to the last data available, only 705 users with reputation between 125 and 250 had casted any downvote (out of 4065). From them, only 33 had casted 10 or more downvotes. Yes, there is an exceptional case, but the rest seems perfectly normal.
In addition, you don't know what posts did these users voted (because votes are secret!!!), you don't know their motives, and you don't know whether these users are in fact knowledgeable new users. You don't really know anything about downvotes, but you're proposing to change the whole system based in your prejudices.
In the technical part, the privilege system isn't controlled by the Physics.SE site, it is a part of the SE engine. In order to propose a change, you have to go to mother meta. And believe me, you will need much stronger arguments.
Yes, this can be a problem, but as Bosoeando showed in his answer, with one exception, it's a very small problem. Keep in mind that the cure to a small problem is oftentimes worse than the disease.
Moreover, the biggest areas where this could be a problem have already been addressed. One potential problem is serial down-voting, which is mostly committed by users with a low reputation. The SE software automatically detects and corrects most of these serial voting problems.
Another potential problem is questions that have made the SE hot list. These hot questions are an open invitation to people from elsewhere on the SE network to join the site. The problem was that users with a high enough reputation elsewhere on the SE network receive 100 points just for signing up with another element of the network. The solution is to quickly mark these hot questions as protected. Newbies who receive those freebie 100 reputation points cannot vote, up or down, on protected questions.