1
$\begingroup$

What's a good ratio for $\frac{\text{Number of declined + disputed flags}}{\text{(Number of flags raised) - (Number retracted)}}$?

Just an approximate... 1:50, 1:75, 1:100 or something rough. It obviously will only be even a little significant when you're considering a large number of flags (maybe >80).

I wanted to judge whether or not I'm generating too many unnecessary flags. But hopefully this can extend to the whole community so people can place how 'good' they are at flagging.

Include close votes as helpful/bad flags for the applicable reason, if you believe it's acceptable to do so: I don't know too much about how close votes and the corresponding reviews work.


I'm aware that it's not going to be very effective as an overall metric because for example, I've had 0 off-topic flags declined/disputed in about 150-170, but a noticeable fraction of VLQ flags disputed. Additionally, most of the disputed flags I raised were during my first couple of days of active flagging, but very few are recent, so the ratio neglects historical issues. It also doesn't show if you've been getting flags declined because you're flagging stuff that shouldn't be flagged.

So this measure really conceals extremely important nuances... But it's reasonable as a rough estimate and can tell someone if there's a huge problem. Technically, a simple modification of the formula can be applied to judge each flag type, i.e. I can use the formula on just the NAA flags I've raised, and then again separately for off-topic.

$\endgroup$
  • $\begingroup$ Mainly not the total ratio of your flags are counted by the system, but the ratio of the helpful/declined flags in the last some months (may 3 or 4 months). If you see warning in the flagging popup, or you are even banned from flagging, it depends on this ratio. But as you are creating the flags, they can become helpful or declined any time, while they are also continuously getting out from this flagging window. The result is that the ratio of your accepted/declined flags can change even multiple times in any direction, without you had raised a single flag. $\endgroup$ – peterh Sep 12 '18 at 14:02
  • $\begingroup$ If you are unsatisfied with your flag ratio, make many reviews, it has a strong positive effect to it. $\endgroup$ – peterh Sep 12 '18 at 14:03
  • $\begingroup$ @peterh I wasn't unsatisfied... I was unsure if I should be unsatisfied. $\endgroup$ – user191954 Sep 12 '18 at 14:39
6
$\begingroup$

So, story time.

Once upon a time, it used to be that Stack Exchange had this thing called "flag weight", which was basically pretty much the very metric you're interested in, and which was shown to moderators (when they dealt with your flags) as well as publicly (because... reasons?). To put it mildly, it was, um, contentious: people started really caring about it and it skewed the system in all sorts of unpredicted ways.

(Basically, every time you put a number next to someone's name online, you create an incentive for them to maximize it (or minimize it or whatever); it's just how humans work. This may be a good thing or a bad thing, depending on what the incentives do, but the design needs to put the stuff that does good and not show the stuff that doesn't.)

So, yeah, flag weight died, and it got completely banished as a metric. Instead, we got a nice dashboard giving us a good in-depth look at our flagging histories, and everyone was happier. (To be clear, some flag information is still public, notably the total number of helpful flags, but no amount of declined or rejected flags will cause any change in public-facing information.)

What's the moral here? Well, for one:

  • do not worry about individual rejected or declined flags.

Seriously. They're not worth it. What should you worry about? Two things:

  1. Moderator messages telling you that you should change a specific flagging behaviour.
  2. A pattern of declined / rejected flags that takes up a significant fraction of your flagging history where all of the flags share rather similar characteristics and were declined / rejected for similar reasons.

Absent either of those, seriously (and I know it's hard to drop it), it's not worth it. Flags are there to bring attention-worthy items to the attention of people who can take actions about them, i.e.

  • the Very Low Quality queue where random nonsense can be sent packing,
  • the Close and Reopen queues where community consensus can be built up for closure or reopening,
  • moderator reviews of rude or abusive comments or spam,
  • direct removal of abusive comments and spam once enough users flag them, and
  • moderator action on anything requiring detailed attention.

In short, there's nothing there that actually does anything, beyond the removal of flagrant spam and abuse if six independent users flag it as such (if I remember the threshold correctly). And, as such, the negative consequences of a few flags going wrong here or there are pretty minimal. If it's a pattern it does need to be fixed, but beyond that, don't worry about it.

$\endgroup$
  • $\begingroup$ Making sure I understood this right: 1) There's no 'good' ratio because you shouldn't be using a raw number to device your flagging strategy/pattern, 2) The thing that is important and should be used to change the pattern is the rejection reason. $\endgroup$ – user191954 Jul 3 '18 at 15:06
  • 4
    $\begingroup$ Interested readers might note that some flag history information is public, and the declined flags ratios for the asker, currently 9/212 = 4% declined, and the answerer, currently 58/1748 = 3% declined, are (with reasonable estimates for statistical fluctuations) exactly the same. Neither is a problem from the moderators' standpoint. $\endgroup$ – rob Jul 3 '18 at 15:06
  • $\begingroup$ @rob I think that's a mod privilege. Opening my profile in a private/incognito tab (not signed in), I can look at the number of helpful flags raised by Chair, but not the number of declined flags. Signed in, I just looked at Emilio's profile, and I can only see the number of helpful flags he raised, and no information about declined/disputed flags again. Clicking the flags link on his profile doesn't do anything, so I don't think regular users (or at least <2k rep) can see the percentage or any similar statistics about declined flags. $\endgroup$ – user191954 Jul 3 '18 at 15:42
  • $\begingroup$ @Chair You may be right. I thought I found a declined flag count on another user's page on a site where I'm not a moderator, but either I used a beta site where reputation limits are different or I just goofed. If either of you would like your non-public declined flag count hidden I will edit or remove my comment. Sorry! $\endgroup$ – rob Jul 3 '18 at 16:00
  • $\begingroup$ @rob Edited to clarify a bit. I'm OK with my declined-flag count being public, though I intentionally didn't include it in the spirit of keeping explicitly away from gamifying that metric. $\endgroup$ – Emilio Pisanty Jul 3 '18 at 16:30
  • $\begingroup$ @Chair That is roughly correct, yes. It's a bit more complicated as for queue-handled flags you won't be given a rejection reason (though I guess if you're really determined you can dig around SEDE and find the review ticket for that post; reviews themselves are public but not easily discoverable). The points where you should get worried is if you get a large fraction of declined / disputed / rejected flags that fall roughly in the same category, as that may point to the fact that there is a pattern you need to change (and then you look at the cases to try to figure out what the pattern is). $\endgroup$ – Emilio Pisanty Jul 3 '18 at 16:35
  • 5
    $\begingroup$ It is also worth keeping in mind that the programatic notion of a "disputed" flag was introduced after the demise of the flag weight, and it is even less useful in some ways: a pattern of incorrect flagging by a different users can "dispute" a host of perfectly good flags that you cast. Indeed, I think the largest fraction of moderator time looking at flagging lately has been spent on "What (or who) is causing all these disputed flag and how to we re-establish a better consensus?" $\endgroup$ – dmckee Jul 3 '18 at 16:42
  • 1
    $\begingroup$ Indeed. Basically, what can happen is that you raise a correct NAA flag (say), and then someone sees that post on the VLQ queue and incorrectly decides that it Looks OK. There, two or even one Looks OK votes can knock the thread out of the review queue, which will mark any original flags as disputed (iiuc). In other words, 'disputed' does not mean that the flag was incorrect. $\endgroup$ – Emilio Pisanty Jul 3 '18 at 16:47
  • $\begingroup$ @rob (several comments up) It's fine, you can leave the comment there. I, like Emilio, didn't want to include my own flag statistics because that would have decreased the universality of the post and made it sound like I was specifically asking about my case. Also, I wasn't completely sure where disputed flags would stand, but dmckee's and Emilio's comments cleared that up. $\endgroup$ – user191954 Jul 4 '18 at 4:14
  • $\begingroup$ Some general information is here. That's from the age of flag weight, but some of the statistics are still relevant: for people who flag actively (>10 flags raised over a week), 10% is an unhealthy percentage of declined flags and it triggers a warning below the flag message. $\endgroup$ – user191954 Oct 7 '18 at 11:06
4
$\begingroup$

I agree with what Emilio said, but just to put some numbers on it: if you're concerned about that ratio dropping below 1/50, you worry too much ;-) It can safely be higher than that. Much higher. Actually, the ratio doesn't matter at all, as far as I'm concerned. Even if your ratio is somehow greater than 1 but the flags you're getting declined are for sensible reasons (i.e. not the same stuff you've had declined before), that's not something I would consider to be a problem.

$\endgroup$

You must log in to answer this question.