I haven't enough experience on Stack to have formed a view about down-voting in general, but as personal testimony I was really put off Stack for quite a while - I've only just returned today - by a down-voting event I was at the receiving end of. I will obviously link to the question I had asked at the bottom, and for that reason I won't give details that are available there, and will assume anyone taking an interest in my answer will obviously want to check my account by looking at the question itself.
What happened was I asked a question that was commented on by a user with significant reputation effectively dismissing the question. The user had already been given a significant ++points for his comment, and my question and been down-voted and stood at -1. When I returned, I pointed out that I was literally quoting (well, paraphrasing) a very senior cosmologist and provided the YouTube link. I also pointed out that my question actually supposed what the cosmologist said had to be false and gave my reasons. So even without clarifying, just in the question itself, there were no grounds at all for being dismissed. This matter got sorted out in the question/answer process, in that the answers all confirmed the cosmologist's position.
All of that was fine. But my negative down-vote was not corrected - even back to 0. Nor were the generous number of points awarded the initial commenter who got it totally wrong.
I felt like this totally invaded and violated my integrity. OK I'm just making that up tee hee.....if an isolated event it's doesn't really matter I can see that. But the reason I suspect it isn't, is because that kind of patter, I would imagine, corresponds to one aspect of the negative dimension to any kind of social reputation system. Which o the whole is obviously a great idea and even if it wasn't, it's just a reality for humans anyway. But there are down-sides that should be well understood and taken serious so as to minimize their effects.
One example is over-trusting people with higher reputation. Not a problem for beginners and casuals...but becomes a potential problem for more senior people, such as those with moderating rights in context of StackExchange, if they have the power to down-vote beginner questions based on high-value users if they comment.
Another example is - and this is relevant to the question above - a kind of statistical tendency for people to play it too safe. Reason being there's a lot of cranks and agendas out there, that aren't really very scientific at basis. So there's a need to keep StackExchange scientific in line with its goals, and that tends to make it necessary to quote consensus positions only. And this then tends to intersect with the reputation system with the result the basis of reputation, the higher up you go, becomes all the more squeezed into very conservative non-risk taking positions. Which leaves a lot of questions unanswered or answered in a way that implicitly rewrites the question to exclude components that speculate beyond what is safe.
I want to really emphasise here, that in hindsight I think leaving stack for weeks over this was an over-reaction. I think the system is really fabulous and I haven't observed a single incident involving a moderator or anyone else that can reasonably be called 'badly motivated' Everyone wants to teach and learn and share and that's terrific.
So for that reason I want to conclude by suggestion a solution based on the above personal experience - which may be biased and probably is so please go look - and personal thoughts about 'reputation' which may be wrong and probably is. Furthermore my solutions, for all I know are already in place, in which case....whatever!
My suggestion is for three new workflow components. Both of these changes assume the good-will and best intentions of moderators and others, which I've observed. It's still down to their personal judgement.
Alter the Stack-exchange application, such that a 'down-vote' action is changed from a simple control event, into a process that begins with the down-voting event, and completes a few days later when the task shows up on the same person's 'to-do' list, to simply review the decision based on subsequent events if any. At that point they can either confirm the down-vote, mark it for second-opinion, reverse the decision, change it, whatever.
Have a new status for questions that defaults to something like "answerable within consensus", but other statuses like "speculative/risky" and then "tagged for deletion". Include an explanation about the reality of the world being full of non-scientific explanations and agendas and the need for stack-exchange to maintain scientific integrity.
add a new status to individual comments/answers that corresponds to the status of the question at the time. So in other words, it becomes possible to give a risky answer to a risky question. It's still going to be more risky than answering within consensus, and there are good reasons why it should be. But it will allow people to take measured risks...in the knowledge that anyone judging will have sight of the context that the question itself was risky. It's still down to personal judgement, how risky is reasonable, and how much is too much.
Concluding remarks
I think changes like this will have a small but compounding effect. Over time and over many many events and people, the impact will continue to grow and eventually it will become dramatic/striking. This is because it directly mitigates the down-side of reputation without diluting reputation. And reputation is fundamental to human society, so the impacts are larger over larger intervals. Thank you for reading.
The promised link:
Why are orbits around black holes stable?