21
$\begingroup$

One of the differences between Physics.SE and Stack Overflow is that SO has at least one review queue which we don't have: Triage. From the 2014 announcement comes this description:

We've been trying to find a more automated way to categorize questions when they're asked rather than requiring each and every question to be moderated.

[...]

Behind the scenes, a "quality score" is calculated for each question based on an automated analysis of the content. Those that score well are sent immediately to the homepage; those that score poorly will now be sent to Triage. From there, they'll go to one of three places based on human input:

  1. The homepage, where they can be answered
  2. The close or moderator flag queue where they can be reviewed and eventually deleted
  3. A new "Help and Improvement" queue where they can be edited

That is, there exists a machine-learning toolset which flags some questions that shouldn't be answered right away, but should be looked at and possibly improved (or closed) first instead. These questions can be reviewed by anyone with 500+ reputation, just like the other review queues.

In a recent Meta post on updates to the review queues, under "Planned changes," appears the statement:

The Triage queue will also be available on all sites but only turned on by default on Stack Overflow. Have a meta discussion and ask a moderator to if your site wishes to add the Triage queue.

The post says that more details about the upcoming changes to the review queues will be available later in Q3 2021, closer to their launch.

Here on Physics, we get a fair number of questions from newcomers and passers-by which are off-topic according to our site's homework policy. While the homework policy was frequently discussed in the past, Meta activity about it has died down in the past few years as our community's consensus has matured. However, we still have a regular trickle of off-topic homework-like questions which accumulate answers before they are closed. Evaluating whether an answer to a homework-like question is "complete enough" to warrant deletion is my least-favorite part of going through the moderator flag queue. It'd be much more fair if those types of questions could be hidden from answerers who don't yet know they are off-topic, so that those answerers don't waste their time on off-topic questions.

So, discuss. Is a Triage queue something that we want on Physics? Attempting to train Stack Exchange's AI to keep low-effort homework questions off of the front page — is that a reasonable goal? How would we decide if it were working like we wanted? What would be pitfalls or problems to beware of?

$\endgroup$
9
  • 7
    $\begingroup$ Kudos to @tpg2114 for having this idea. $\endgroup$
    – rob Mod
    Jul 3, 2021 at 1:34
  • 1
    $\begingroup$ This is only a question at all to people who think that SE's rules and ethos are in some sense "important". Clearly the people who ask and answer HW-type questions do not. Whether that is from ignorance or from deliberately ignoring the rules is beside the point - the only point is that SE is not providing what that subset of its users want. (My personal view is that the objective of SE to be a "global repository of excellence" is frankly ridiculous - and I've seen plenty of other failed attempts at the same thing on the internet, started by people with more ambition than common sense.) $\endgroup$
    – alephzero
    Jul 3, 2021 at 13:42
  • $\begingroup$ "Attempting to train Stack Exchange's AI to keep low-effort homework questions off of the front page — is that a reasonable goal?" - My understanding was that the system would learn from the results of the reviews automatically, but the way you wrote this might imply that someone would have to sit down separately to give it a training set. It's the former, right? In which case, I don't see what the burden would be over what's happening now. If it's a big separate job to get it started, then the cost might matter. $\endgroup$
    – Brick
    Jul 3, 2021 at 22:22
  • $\begingroup$ @Brick What I have written here is somewhat more than 100% of what I understand about the Triage queue. I have the idea that, as part of enabling the queue, there might be a trial period where we could see what the algorithm flags without it actually keeping things off of the front page. But at this moment I don't recall whether that's a real feature I read about, or something that happened on SO in the distant past when this feature appeared there, or an idea that I had on my own with no connection to SE's capabilities. $\endgroup$
    – rob Mod
    Jul 4, 2021 at 2:09
  • 5
    $\begingroup$ The triage queue is and has been an integral part of stackoverflow as long as I’ve been part of it. Checking the network profiles of those who seem to think this wouldn’t be useful so far, I notice little or no activity there - not enough for activity in the triage queue. As a person who has done more than 1,100 triage reviews, I think this is the single most important review queue of them all. Will try to formulate that into a full answer soon. $\endgroup$
    – Brick
    Jul 4, 2021 at 2:13
  • 3
    $\begingroup$ It would be good to have a definitive statement regarding whether an incomplete (or failed) Triage review would keep a question off of the home page. $\endgroup$
    – Emilio Pisanty
    Jul 5, 2021 at 18:11
  • 1
    $\begingroup$ @EmilioPisanty See this 2018 post which refers to Triage as “a pre-filter for the home page.” That post points out that, on SO, most questions aren’t found directly from the home page, but presents data claiming that questions which leave Triage as “unsalvageable” still have fewer average views than questions which leave Triage as “looks good.” $\endgroup$
    – rob Mod
    Jul 6, 2021 at 2:14
  • 3
    $\begingroup$ Since this feature (for network-wide release) is still in development, we can't add it here at this point in time, thus the status-deferred tag. That said, considerations like those proposed in Brick's answer are things that are great to hear about - we think that hiding questions that are in Triage from the front page is a great idea and will address a lot of the issues that some sites experience with new questions. While it's not what we do in general because it's not usually necessary on SO (since the front page is so active), we should consider it as part of the network-wide feature. $\endgroup$
    – Catija StaffMod
    Jul 19, 2021 at 17:22
  • 1
    $\begingroup$ This was referenced in the 2021-10-09 MSE post Help Center still says that Triage is "Stack Overflow only"; however, it's enabled on Physics since September. $\endgroup$
    – Peter Mortensen
    Oct 10, 2021 at 15:38

4 Answers 4

Reset to default
14
$\begingroup$

I went into Stack Overflow and handled a few questions from the Triage queue there to remind myself a bit how this works. I've done over 1,100 triages on SO, but it had been a while so I had forgotten a few details myself.

As background, if you're not familiar with this queue, you might want to skim the workflow here: https://stackoverflow.com/help/review-triage

Now a couple of misconceptions that I think I see in both the question by rob and some of the answers:

  • Rob framed this in terms of the HW policy on Physics.SE, but we should be clear that there's no reason to think that questions entering Triage would necessarily be HW questions only or that all HW questions would get sent there. The behind-the-scenes scoring would likely catch other questions and miss some that are definitely HW.
  • The Triage queue is about questions and not answers. The system does not put answers into this queue at all.
  • Questions in Triage are not necessarily (ever?) held off of the homepage while in the queue. The material that rob quoted suggests that they are - and maybe it is true in some way - but I tested that I can access questions in the queue directly on SO without going through the queue. It also happens that some of the questions in the queue already have comments and even answers. (To be explicit, I pasted the URL of a question in the queue into a new browser tab. So that's not the same as finding it on the homepage.)
  • For questions heading to closure, there is no extra work incurred here by the community. If I select one of the top-level options in the review that leads toward closure, I end up casting a close vote directly from this review. For questions that are OK, it does look like this might add a couple of reviews, since the material that I read indicated that it takes 3 "Looks OK" votes to clear it.
  • This queue has a companion queue on SO for questions that are marked for edit by community (rather than edit by author) where they go for "help and improvement". It wasn't clear if we'd get that queue too if we enable Triage. For those concerned that this queue is strictly punitive, however, this option might appeal.

With all that as background, the question here is still whether or not Physics.SE wants to turn this on. My initial reaction was that we should, but that was based in part on the apparently wrong belief that items heading into this queue would not immediately go to the homepage.

If it were true that items in the queue would not go to the homepage until cleared, then I think this would be a definite improvements in these respects:

  • It would reduce or eliminate the incentive for people to post poor questions in the first place, most especially the egregious HW ones that demand an answer in a very short time.
  • It would reduce or eliminate answers to poor questions, including some HW questions, if they could be closed before they got posted to receive answers. This would have secondary benefits of facilitating roomba clearing these questions out automatically and reducing the number of flags going to the moderators to deal with HW answers.

Assuming that it's not true that the questions are held though, I'm inclined to think that enabling this queue will not help:

  • On SO, where the question volume is high, it helps ensure that experienced users look at all of these questions, so there's value there even if the questions are on the homepage.
  • Physics.SE has lower volume and I suspect that there are already a handful of users who eventually look at every or nearly every question that's posted.
  • Ultimately the main problems that I see directly or most discussed arise from lack of people (in total and per unit time) participating in the review queues that we have and/or using the downvote according to its SE purpose. For a few who are particularly active in the queues, one might argue instead that it's the 20 review-per-day limit that's the problem in their cases. Adding a queue isn't going to address those problems. It probably won't make it worse, but I don't think it can help - unless it comes with a hold on the question being posted.
$\endgroup$
7
  • 1
    $\begingroup$ I understood that this would go to a moderator queue; is this queue accessible to all or only moderators? I agree that keeping low quality questions off the home page is a very nice feature; one of the strengths of a site like this is that low quality questions are removed by some measure of consensus, and only very rarely by individual users (either mods or gold-badged users), so I wonder how much the triage erodes this if it’s limited to a queue accessible by mods only. $\endgroup$
    – ZeroTheHero
    Jul 4, 2021 at 12:58
  • 1
    $\begingroup$ @ZeroTheHero it’s accessible to users with 500+ reputation, like the other review queues. $\endgroup$
    – rob Mod
    Jul 4, 2021 at 13:07
  • $\begingroup$ @rob ah! ok then. I had misunderstood that part of your post. $\endgroup$
    – ZeroTheHero
    Jul 4, 2021 at 13:17
  • $\begingroup$ @zerothehero So to be clear, there are actually more users with access to this than, say, close reviews. If your rep is high enough then when you choose the “flag” option in Triage review, you actually end up casting a vote. If your rep is not quite so high, then it raises a flag. But a close flag, I believe just puts it in the close review queue, so that’s still not just for the mods and an option the same user would have had if the clicked not the question on the homepage. $\endgroup$
    – Brick
    Jul 4, 2021 at 13:57
  • $\begingroup$ Not sure if it's helpful, but there's more and current elaboration at meta.stackoverflow.com/questions/295650/…. $\endgroup$
    – Andras Deak
    Jul 6, 2021 at 12:19
  • $\begingroup$ I don't quite follow your "cons" points: (1) there might be some users how eventually look at every question, but is relying on those a good idea? At the very least, having other people also look and flag potentially low-quality questions seems like a good kind of redundancy to me. It meanas more hardcore users will have to worry less about missing some question that ought to be closed immediately and such. (2) Why would the other cue be detrimental in regards to the close vote limit? If you have enough rep for the regular cue, you don't need to touch the triage at all, no? $\endgroup$
    – glS
    Jul 11, 2021 at 9:39
  • $\begingroup$ I'd be tempted to say that the main advantage of triage would be the significantly larger pool of people that can use it. That having a potentially positive effect both because of their direct help in managing closing questions, but also for the fact itself of engaging more people in site maintenance. This might also encourage more users to engage in regular close voting later on. Granted, I don't know if this actually happens; one would need data to tell whether having a triage ends up increasing the chance that a user will actively engage in cues once they have enough rep $\endgroup$
    – glS
    Jul 11, 2021 at 9:45
5
$\begingroup$

This feature has been enabled on September 9th; this was the first review in the new queue, and the first one in the network outside Stack Overflow.

$\endgroup$
3
$\begingroup$

Evaluating whether an answer to a homework-like question is "complete enough" to warrant deletion is my least-favorite part of going through the moderator flag queue.

I don't envy the moderators this task. Having to do crap like this is one of the reasons I have never wanted to become a moderator. I don't mind voting to delete when I have four other voters to hide behind but I would find it stressful if mine was the sole responsibility.

Although it grates against my OCD tendencies I think the best strategy is probably to just ignore the answers to homework questions. In the main they are from new members who are only trying to be helpful. As long as the question gets closed reasonably quickly I'm not sure it matters whether it gets answers. If an OP is abusing the system by repeatedly asking homework questions the SE's auto-ban algorithm will eventually get them.

I am amongst the more determinedly anti-homework elements of the site membership, but when it comes to eliminating the homework questions we don't need to be perfect as long as we are good enough - and I think we are good enough. So I don't believe the triage system is necessary, though I have no objection to it if others feel it is worth while.

This doesn't help the mods who have to deal with the flags, but you could relieve the stress by simply rejecting most of the flags unless you're seeing the same user names coming up repeatedly.

$\endgroup$
7
  • $\begingroup$ I tend to agree that a star chamber would be perceived negatively. In view of this, what are the advantages of triage over implementing the 3-votes-to-close proposal that has been discussed previously? $\endgroup$
    – ZeroTheHero
    Jul 3, 2021 at 13:23
  • 1
    $\begingroup$ @ZeroTheHero I don't think the three-vote closure was ever declined. I think we are just waiting for it to be our turn to try it out. (I'm not in a place to look for the relevant meta thread though) $\endgroup$
    – BioPhysicist
    Jul 3, 2021 at 16:30
  • 1
    $\begingroup$ @BioPhysicist I would prefer a 3-vote-to-close policy, if only narrowly applied to HW questions. $\endgroup$
    – ZeroTheHero
    Jul 4, 2021 at 0:54
  • 1
    $\begingroup$ I tried hard to make this proposal not about my own distaste for dealing with homework-like questions. My main concern is for users --- especially new users --- who invest time in writing quality content that is destined to be downvoted and/or removed; my dislike for participating in that process is secondary. Your choice of pullquote suggests I was unsuccessful in making this proposal not about me. (My therapist assures me that it's okay when people know about my feelings, so I'm not going to attempt any sanitizing edit.) $\endgroup$
    – rob Mod
    Jul 4, 2021 at 1:56
  • $\begingroup$ As noted in my answer, the focus on HW is a bit of a red herring and should not dominate the discussion or the decision to enable Triage. If enabled, Triage will start picking up all questions that it thinks requires attention - that may or may not include HW questions as we define them. $\endgroup$
    – Brick
    Jul 4, 2021 at 4:18
  • $\begingroup$ @rob maybe then broaden this queue beyond the moderators; 10k users have access to moderation tools and 20k users are “trusted” so they could participate and dilute the workload and share the burden. $\endgroup$
    – ZeroTheHero
    Jul 4, 2021 at 12:49
  • 3
    $\begingroup$ One reason to not ignore answers to homework questions is that a positive scoring or accepted answer blocks the roomba from deleting the question. But I guess that's not an issue if mods / high rep members are prepared to manually delete those questions. $\endgroup$
    – PM 2Ring
    Jul 4, 2021 at 17:49
3
$\begingroup$

Do we really need this ?

Don't get me wrong - it's a tempting idea, however ...

The existing system does provide at least one benefit : it lets more experienced users demonstrate to newer users how the homework policy works. I think it is important that new members (not just passer-by types) see how this works in practice.

If a lot of magic happens behind the scenes we're actually hiding a member policy as well. Remember, this is not a policy written in stone. It could change. It seems to have recheached a consensus now, but those change over time. It's going to be a lot harder to change an AI to respond to finesses or even complete changes in the homework policy. Not all the science orientated sites have a homework policy, so that's also an issue.

I think we would be shifting the workload from the general membership to either moderators or those who actually do check the review queues (sense of guilt there :-) ). I suspect that makes the existing system more practical.

Humans, those all handy general purpose machines, will tend (I think) to do a better job of working out what is and is not homework than some machine. Particularly as the "adversary" here is also human and they're bloody devious, these humans. If we're going to have to triage them anyway, let's at least be visible about it and show new member the why and what of the policy and give them a feel for what is black, white and grey.

And then there's Terminator to worry about. Say no to AI before it's too late. :-)

$\endgroup$
9
  • 4
    $\begingroup$ I don’t follow your logic here regarding shifting workload. If the triage keeps HW questions from showing on the webpage until they are closed, then we don’t get answers to them. So then nobody is flagging HW answers to the moderators as they do now. For the general users working the review queues, this is exactly one more total review by the community - the one in triage. Because after that it goes to the close queue (same as now when it’s eventually flagged) or to homepage (same as now). Maybe I’m missing something but this seems like an obvious thing to enable. $\endgroup$
    – Brick
    Jul 4, 2021 at 0:00
  • $\begingroup$ John Rennie did suggest we could do just as well to ignore homework answers. I don't think flagging the moderators is really useful for answers. We might be better to just let members gently remind users that they are answering homework and direct them to the homework policy. What does flagging answers achieve, really ? Many homework questions are borderline - preventing answers is counter-productive in those cases, IMO. $\endgroup$
    – StephenG - Help Ukraine
    Jul 4, 2021 at 0:16
  • 1
    $\begingroup$ Flagging them and having them removed, depending on a few factors, makes the question & other answers eligible for roomba. One may argue about how much people care about that, but it is a tangible difference. Declutters the site, removes incentives for asking & answering HW. $\endgroup$
    – Brick
    Jul 4, 2021 at 0:23
  • $\begingroup$ That aside, I remain genuine confused about your post as the argument seems to lead logically to the opposite conclusion from the one you’ve drawn. $\endgroup$
    – Brick
    Jul 4, 2021 at 0:26
  • 1
    $\begingroup$ @StephenG If we ignore policy violations then what is even the point of the policy? $\endgroup$
    – BioPhysicist
    Jul 4, 2021 at 3:17
  • $\begingroup$ @BioPhysicist I didn't say ignore that it's homework and violates policy. No idea why you think that. $\endgroup$
    – StephenG - Help Ukraine
    Jul 4, 2021 at 3:19
  • $\begingroup$ Sorry, I meant about the answers $\endgroup$
    – BioPhysicist
    Jul 4, 2021 at 4:45
  • $\begingroup$ @BioPhysicist Understood. I'm simply suggesting that (a) answers to HW question are not necessarily wrong. Some answers are OK, and what is HW is often a grey area. Maybe worrying about the answers is more bother than it's worth and we just rely on members to educate the offenders without making a crime out of it. Punishment maybe is for persistent offenders - then bother the mods with a flag. $\endgroup$
    – StephenG - Help Ukraine
    Jul 4, 2021 at 4:56
  • 2
    $\begingroup$ "Punishment maybe is for persistent offenders - then bother the mods with a flag." But how is someone to know if a user is a persistent offender? The only way to do that would be to dig through their other answers. Plus that just adds another layer of "education": "Oh, answers to homework aren't allowed? But then why am I seeing other questions with answers?" ... "Welcome to PSE! Answers to homework aren't allowed, but you can answer homework up to X amount of times without it being an issue." It just adds another layer of subjectivity. It sounds like a mess to me. $\endgroup$
    – BioPhysicist
    Jul 4, 2021 at 11:14

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .