Flagging low quality answers: an experiment

Question

Abstract: We are experimenting with expanding the scope of the very low quality (VLQ) flag. For two weeks, August 8-21, moderators will not decline any VLQ flags. We will allow these flags will sit in the queue as long as needed for the community to deal with them. We ask you, the community, to

• use VLQ flags on posts which are "harmfully incorrect" (see below for what that means), which would ordinarily not be a valid use of the VLQ flag
• when reviewing low-quality posts, choose to delete such "harmfully incorrect" posts

---

The handling of low-quality answers has been a point of contention and friction, on and off, for a fairly long time in this site. For low-quality questions, it happens that every so often someone sees a terrible answer, flags it as low quality, and finds out that it gets declined with the message

flags should not be used to indicate technical inaccuracies, or an altogether wrong answer

which then generally leads to bruised egos and a lot of friction, and also a lot of people unhappy that we don't have a strong enough mechanism to handle the types of posts they wanted to flag. (Examples: 1, 2, 3, 4, 5, 6, 7, 8.)

This thread is an attempt to explain what pressures the system is under, the way the flagging software works under the hood, the way flags are currently handled, and the reasons for this. I will then make two proposals for how the system could be improved (one of which we can implement ourselves, and the second of which would include software tweaks to support that workflow), and I will propose an experiment to see how well it works.

I worked out several of these aspects after going through the bruised-ego, unnecessary-friction cycle in Why was this Very Low Quality flag declined?, and the responses and votes on that post and in a related chat make me hopeful that there's appetite for a change and a rough agreement that it's possible.

---

Summary

In short, I claim that answers that are (1) flat-out incorrect, (2) so completely incorrect that no amount of editing can make it useful or constructive, and (3) so completely incorrect that it is actively harmful to the site and its standing, should be removed from the site. They should be flagged as Very Low Quality, and should be deleted via a consensus of six 2k+ users or three 20k+ users.

For the next two weeks, let's experiment with this dynamic and see how well it works.

---

The pressures

This site gets its fair share of posts, both questions and answers, that are - shall we say - rather below par. This includes a lot of content that a lot of people would rather we didn't have on the site at all and, in particular, all sorts of content drawing from every depth of crackpottery.

• For example, we get a lot of questions that assume as a starting point some way-off-standard theory and then (some question or other), in such a way that the only possible responses would either be a complete rejection of the hypotheses, or fringe pseudophysics with no place in any serious physics forum.

• We also get a steady smattering of answers that draw on such off-base foundations, and which look something like "time dilation does not actually exist because Einstein was wrong, see my site at for more explanations".

  The two types of post above are usually made worse by the fact that in a large fraction of cases the poster will adopt rather belligerent attitudes, refuse to back down on any claim or listen to anyone at all, and very often drag the discussion down to wildly unconstructive mud-slinging, which is unbelievably draining for everyone involved.

• In addition to the above, we also get a steady stream of posts that are just not very good. This ranges from incomprehensible keyboard bashing, to one-liners that only vaguely address the question, one-liner questions with nowhere near enough information to be answerable, 'answers' that are just comments on another post or which ask a separate question, and so on.

(This is, of course, an incomplete list.)

Luckily, of course, the site has a bunch of ways to deal with these things. For off-the-field questions we can close as non-mainstream and we can downvote them (which, once they reach score -4, hides from the active-questions list). For 'answers' that are really a separate question or a comment we can flag as Not An Answer, and it generally gets resolved rather well relatively quickly. For gibberish posts and one-liner nonsensicals we can flag as Very Low Quality and they eventually disappear.

For the very-much-wrong, out-in-the-out-field-of-the-next-field-over answers, we can downvote and... that's it. The way the system works now, there is often precious little that a user can do if she notices a post that is wrong-wrong-wrong. (Think something like this or like this.) As long as it is an attempt to provide some sort of answer, the Not An Answer flag doesn't work, and the Very Low Quality flag currently excludes answers which are generally intelligible.

Our user can then add a comment explaining what is wrong (with the associated risk of getting dragged into an unconstructive discussion), and she can add a single downvote, which often looks like a hopelessly small signal (like, for example, with answers that might get upvoted in a popular question because they 'sound right', or are catchy enough, to enough outsiders, even if any working physicist can see that they're completely incorrect). There isn't even a way to get other experts to notice the post and help fix it.

This, I contend, is a problem.

What happens to flags once you flag them

The way the system is set up at the moment, there are two moderation queues where a post goes after it is flagged: the Low Quality Posts community moderation review queue, and the moderator queue where all flags end up when they require human intervention to get swiftly handled.

So what does the Low Quality Posts queue do?

• For questions, the queue looks something like this:

  

  It enables the reviewer to close the question if it is off-topic, edit it if it can be whipped into shape, skip it if the reviewer doesn't know what to do, and decide it looks OK and shouldn't really be in the queue to begin with.

• For answers, the queue looks something like this:

  

  It enables the reviewer to edit the post, help get it off the queue by saying it looks OK, skip it and move elsewhere, and -- depending on the poster's reputation -- recommend deletion of the answer (<20k rep) or a vote to delete it (>20k rep). Deletion votes come with the option of posting a canned message that includes a link to the review page.

Posts with a negative score get deleted when they get six Recommend Deletion votes, or three Delete votes by >20k rep users. (Posts with positive score with that many delete votes get referred to moderators as a disputed deletion, and they probably do deserve that.)

The Low Quality Posts queue is also populated by automatic processes, which generally get marked by text along the lines of 'this answer was automatically flagged because of its length and content', and includes posts which are very short, have lots of links, are by new users, or probably some combination of those. Generally speaking, the algorithm that catches these is pretty well tuned, and it very often does flag up posts that do require some extra attention.

Posts stay in the queue for as long as they need to: until they're kicked off the queue by having more Looks OK votes than flags and (recommended) deletion votes, the queue produces an action on the post (closure, deletion, or edit), or a moderator intervenes.

In addition to this, after fifteen minutes of being raised, Very Low Quality flags also appear on the moderator queue, where moderators can see them and act on them. (However, this time can be changed and indeed it's a one-hour wait in several network sites.)

How flags are currently handled

The current guidelines for how flags should be raised on this site are at Which flag do I use for an inappropriate post?; in addition, there is network-wide guidance on how to review over on the mother meta at What are the guidelines for reviewing?. In particular, the guidelines for raising a Very Low Quality flag currently read

Very Low Quality

The "very low quality" flag is for answers which are complete nonsense, such as

• incoherent gibberish
• anything not parseable as English text
• answers posted in another language (although if you can edit in a translation, you're encouraged to do so instead of flagging)

Do not use this flag for

• wrong answers
• answers based on non-mainstream physics
• followup questions
• comments

In particular, answers which are wrong or based on non-mainstream physics are explicitly excluded. The reason for this is that Very Low Quality flags very quickly make their way to the moderator review queue, and moderators are entirely the wrong people to be handling this sort of flag. Removing an answer because it's wrong, or because its premises are off-mainstream, is a judgement call based on technical aspects of the post.

This is precisely the sort of task that we use community moderation for - a consensus decision based on the judgement of multiple experts who have been on the site long enough to know what works and what doesn't - but if we ask moderators to handle these flags we're asking them to delete users' content based on technical grounds that they might get wrong. To their credit, the site moderators here understand that this is not their job and they repeatedly refuse to delete posts on those grounds.

In the end, this means that when people raise a Very Low Quality flag on a post that's dead-wrong it ends up being declined by a moderator because the system is asking them to do more than they should be doing. (Comments, of course, are another story.)

---

What should be done with low-quality answers?

OK, so the story so far is how the system works, why it works this way, and what problems this causes. So, in an ideal world, how should this sort of post be handled?

One view which is held relatively widely (example) goes something like

Technically, we can't delete things which answer the question (even if they're wrong or short/low quality).

I contend that this is wrong: we can and should delete answers that are incorrect beyond saving.

(If you disagree, then as I argued previously: what compels us to host content that is wildly incorrect? Say, things like this or this. Users permit Stack Exchange to display their CC BY-SA-licensed content, and they ask the site to display it, but what compels us to host incorrect content?)

To be a bit more precise, I contend that if an answer is

1. flat-out incorrect,
2. so completely incorrect that no amount of editing can make it useful or constructive, and
3. so completely incorrect that it is actively harmful to the site and its standing,

then it should be deleted. In particular, it should be deleted via consensus-based community moderation, rather than through unilateral moderator action.

Stack Exchange sites shine because when you land on a question you're likely to find a good question with an easily accessible good answer written by someone who knows their stuff, instead of pages of back-and-forth trying to get the asker to provide enough information, endless discussions over technical details, or off-the-wall posts that are just plain wrong. In the words of ACuriousMind

the mission statement of StackExchange is "The best answers rise to the top", not "Even the worst answer is here for your convenience".

Or think about it this way: if [insert rising-star researcher in your field] were to stumble on that post on the site, would they think "huh, this is an interesting place" and decide to participate, or would they give it up as a wash because incorrect content gets such an easy pass?

Fixing the VLQ flags, part 1: community process

Hopefully I still have everybody on board (and if you disagree with the above section, I do want to hear your views and I hope you're comfortable sharing them), in that there is a problem to be solved that the current mechanism isn't solving. Luckily enough, though, we do have the software tools to build a community process that can help clean up by putting deletion-worthy content in front of the people who should be deleting it (a consensus of experienced users) rather than the people who shouldn't (unilateral action by a moderator).

I propose, therefore, to change the community guidelines for flagging and reviewing:

• For flagging: use the Very Low Quality flag for answers which are complete nonsense, such as

  • incoherent gibberish
  • anything not parseable as English text
  • answers posted in another language

  as well as answers that are wrong, or based on non-mainstream physics, that you think are

  1. flat-out incorrect,
  2. so completely incorrect that no amount of editing can make it useful or constructive, and
  3. so completely incorrect that it is actively harmful to the site and its standing.

  Do not use this flag for

  • followup questions
  • comments

• For reviewing: vote to delete / recommend deletion on posts that in your judgement satisfy the above criterion. If you're not sure, click skip; if you think the post belongs on the site, click looks OK.

Very Low Quality flags should go to the Low Quality Posts review queue and remain there: that is, mods should not act to decline or dismiss VLQ flags raised over the technical quality of posts (whether they end up in the moderator queue or not), unless the review queue is obviously indecisive about it or is taking overly long to handle a post. I would fix the timescale for this at 2 to 3 days (but this should be easier to decide after the experiment below).

Back-end changes

The above proposal has the enormous strength that it works completely within the existing Stack Exchange software, which is good because asking for developer time to work on something on this scale requires some pretty strong arguments that it will really help the site.

That said, to help it work better it would really help to keep Very Low Quality flags from being shown to moderators for a rather longer time than they do now - substantially longer than an hour, at the very least. This gives the community moderation space to do its work, and refers to the moderators the stuff that can't be handled that way.

This does have the disadvantage that gibberish and other clear candidates for deletion take a bit longer to clean up than they do in the current system. I think this is an acceptable trade-off, though.

Once the community moderation process is up and running, we should also consider other cosmetic changes (like, for example, a re-write of the text in the flagging prompt), but the way I see it these are secondary to the change in the process itself.

---

The experiment

Finally, how do we get from here to there? I propose we experiment by switching to the system described above for two weeks and seeing how well it works. Thus:

• For the next two weeks (Monday 8 August to Sunday 21 August) moderators will still see Very Low Quality flags but they will only act on gibberish and non-parseable text, leaving any calls that depend on a judgement call on the technical merit of a post to the Low Quality Posts review queue.

• During that time, flag and review as per the above guidelines. If you see content that you think is so wrong it's harmful, flag it for removal. If you review content that you think is so wrong it's harmful, recommend its deletion.

After that, we'll have a look at as many statistics as we can get our hands on (some of which do require some data digging by an SE community manager) to see how well it worked.

---

Coda

Finally, if you disagree with the above (including the case where you think we shouldn't even run the experiment) then I am interested in why you think so. However, there is one thing I do want to ask people not to fall into, and that is 'answer by Stack Exchange say-so'.

One example of this is here, which sort of goes "well, Stack Exchange put up these vague text messages, so therefore this is how we should run our own community moderation". Another example is this comment on that same question, coming from an outsider to this site and just referring to generic SE guidance. If you do want to just refer to generic guidelines, then I do ask that you provide a suitable argument for why they're also sufficient to help this site deal with the pressures it's under.

The reason for this is that the Stack Exchange software and the core guidelines for community moderation need to satisfy the needs of a huge array of 150+ sites with a wide variety of communities, attitudes, situations, and constraints.

The engine and guidelines need to handle the enormous volume of Stack Overflow, with review queues so long that people resorted to robotic reviewing so much that review audits needed to be introduced. They need to handle The Workplace, which every so often draws in a crowd of outsiders that's pretty sure it can solve everybody's problem. They need to handle the happy atmosphere of TeX.SE, which is just happy to throw upvotes pretty much anywhere. They need to work for movies and SciFi and RPG, which inhabit a very different corner of the internet than we do.

In particular, for many of those sites, the concept of an answer being 'harmfully incorrect' doesn't even make sense. What does a non-mainstream answer look like in Mathematics? Do they get them in Chemistry or Photography? Unless the guidelines you want to quote were built considering an explicit analogue of non-mainstream, harmfully-incorrect answers, then repeating them as-is is simply not helpful.

Final question: isn't it time I piped down? Why yes, yes it is. Over to you.

Answer 1

I heartily support Emilio's proposal, but to explain why I support it will require a bit of rambling. So please bear with me and I'll try not to make this too painful.

The Physics Stack Exchange has evolved into an educational resource. We hoped it would be a research tool akin to the MathOverflow but for whatever reason that didn't happen. It appears that the culture amongst research physicists just doesn't lend itself to this goal. The Theoretical Physics SE got canned due to lack of traffic and the Physics Overflow is withering on the vine. So an educational tool is what we have ended up with.

But if the above sounds a bit downbeat let me emphasise just how important I think the Physics SE is in its role as an educational tool. Over the five years (good grief!) I've been a member I have explained basic principles of (for example) special relativity to students who have gone on to do PhDs, and it's hugely rewarding to think I might have contributed in some small way to the next generation of scientists. Most of us would, at least in principle, like to make the world a better place and this is one way you can really do it.

But, and it's a big but, if you're going to take on the role of teacher then you have the responsibility of ensuring you teach the truth and only the truth. If a student repeats misinformation that you've given them in an exam, and fails as a result, then you deserve to lie awake at night feeling guilty.

The point I'm working my way round to is that given our role as an educational resource it is important that our answers here are unimpeachable. This is not a place for trotting out your preconceptions and personal biases. Any answer posted here should not raise eyebrows if it appeared in a mainstream physics textbook. Well that's a grand goal, and of course one we are bound to fail at as long as answers are written by humans, but that's the goal to strive for.

And I think we are pretty good at the peer review process. Not perfect of course, but on the whole I think we do a decent job. We spot answers that are confused, misleading or plain wrong and indicate by comments and downvoting that the OP should not trust the answer (not if they want to pass their exams that is). The point of Emilio's proposal is to make this process still better, and that's why I support it.

Answer 2

I do not see where you have provided a definition of the term "harmfully incorrect." I very much doubt such answers exist. I think those who are advocating the view that some answers are so incorrect that they are positively "harmful" are not acting scientifically, since no evidence of such "harmfulness" has been provided. It seems to me that the "harm" is imaginary.

I disagree with John Rennie when he says that Answers on this site should be "unimpeachable". That is setting too high a standard which even those with the highest academic standing in this community may not consistently reach. It is an elitist view which would require most of the Answers on this site to be deleted, and would deter many users from participating by offering an Answer.

The current peer-review system of commenting on mildly incorrect Answers, and "down-voting to oblivion" those which are "harmfully incorrect," seems perfectly adequate to me.

Science tolerates alternative views, thrives in a climate on honest debate, and is stifled by the censorship of anything unorthodox. The fact that Answers are not mainstream or not wholly correct should not prevent them from being expressed. The advantages of maintaining freedom of speech outweigh any illusory "harmfulness".

Answer 3

In short, I claim that answers that are (1) flat-out incorrect, (2) so completely incorrect that no amount of editing can make it useful or constructive, and (3) so completely incorrect that it is actively harmful to the site and its standing, should be removed from the site.

Who says they're flat out incorrect? You? What if your answers are flat out incorrect? What if you're asking for is censorship of correct answers so that your incorrect answers have no competition? This proposal would allow physics stack exchange to be hijacked and censored by a "cabal of quacks" working in concert.

So: no. If there are very-low-quality answers, you explain why they're very low quality. You hold them up to your own answers in comparison. You do not censor the competition with some vague insinuations that that's not mainstream. Not when answers like this get twelve downvotes despite the multiple references to Einstein. Not when what's actively harmful to the site are self-appointed "experts" who are sneering and scathing to newcomers and juniors, and whose arrogance is only exceeded by their ignorance.