Data collection: When should (or shouldn't) we exclude questions from the Hot Network Questions list?

Question

The SE team has made several promising changes to the "Hot Network Questions" system. If you have previously found HNQs to be problematic, you are invited to participate in the discussion over there.

The purpose of this discussion is a new moderator superpower:

Moderators have the ability to remove questions from the HNQ List.

There are times when the hotness formula selects a question that a site would rather not have featured. Up until now, the only recourse that was available was to close the question (which may be appropriate anyway but isn't ideal when done purely to manage traffic), or to do nothing. We're putting the power in the hands of our moderators to remove questions that don't set a good example for their sites. I recommend each site have a meta discussion with guidance for moderators about when - if ever - a question should be removed.

I'd like to propose that we not invent a new policy on these questions de novo. I would prefer instead that we think as a community about the kinds of standards that we already have in place and communicate via voting, close-voting, and flagging, and how this new tool fits into our kit.

I'm imagining a process like this:

1. Over the next few weeks you will, like usual, notice some unusually active questions and think "well, that shouldn't be HNQ."

2. When you notice this, raise a custom moderator flag on the question and try to articulate what your reasoning is.

3. The moderation team will deal with your flag promptly --- that's the point of flags. But we'll also save your reasoning someplace, and as we start to get enough flags to divine a pattern, we'll try to summarize it either here or in a new post.

You could also link to posts in answers to this question, and explain how you think that the Hot Network Questions list has affected them. Those kinds of answers shouldn't be limited to bad posts that should be un-HNQ-ed: part of deciding what our policy should be will be deciding what we do want to keep. I'm especially interested in articulating, as a community, what types of posts get better responses after exposure via HNQ.

I also suggest flagging (or linking) current questions, because there will be some temptation to base any policy we develop on remembered examples from the past. For deciding how we think the site ought to work, it's usually better to notice how we are responding when we are using the site organically. I would hate for us to have a contentious discussion about a problematic question from the past and lock ourselves into some "policy" based on behaviors that don't really occur on the site any more. That's why I've titled this post "data collection": I'm proposing that we spend a reasonable amount of time just looking for patterns, and try to come to a conclusion around May-ish.

---

Update 2022: To see what our community keeps or removes from the HNQ in practice, consider this Data Explorer query.

Answer 1

Questions about perpetual-motion machines should meet a stricter-than-usual bar in terms of clarity and prior research to get on HNQ.

Questions about perpetual-motion seem to have a particularly easy time hitching a ride on the HNQ list (this is one recent example), and it's easy to see why: they are instantly appealing to a lot of people, they are generally easy to understand, and they often serve as ways to either make fun of OP (for being "dumb enough" to ask such a question in the first place) or to rail at the physics "establishment" for insisting on trivial such things as having a consistent physical theory of nature.

However, they get very tiring, very fast. Basically, the only questions on the subject that make the bar for being on-topic are variations on the theme of

I came up with this device and I analyzed it in (insert flawed analysis that concludes with perpetual motion). Where did I make a mistake?

and after seeing two or three of those on the HNQ as Representative Questions of the site, they start to grate very fast. (And, as with the physics-in-fiction vein, they very quickly become a skewed presentation of this site's subject matter.) Frankly, I think we could do with a heck of a lot fewer of those.

I'm not quite sure I'd go as far as proposing that perpetual-motion machines should be completely off-limits for this site's HNQ output, but I would say that for such questions to remain on the list they should be asked in exceptionally clear way, they should have a very clear physics through-line, and they should clearly demarcate themselves from previous iterations of the subject on this site.

Answer 2

OK, I'll start.

Questions about physics in works of fiction shouldn't be on HNQ.

This has been a point of contention for quite some time, and it has a strong track record of leaving people confused and upset, even several years after the question has had its heyday (example, example, example). I'm referring, in particular, to questions like "Could Sam have pierced Shelob's carapace?" or "Could Legolas actually see that far?", particularly when the titles are that explicit in their mentions of the fictional context.

To be clear, this is independent of the decision on whether such a question is on- or off-topic here, which can have both positive and negative answers. As examples, "how does the warp drive in Star Wars work?" is clearly off-topic, while something like "is orbital mechanics being portrayed correctly in (insert sci-fi story)?" has a much better chance of being a legitimate question about physics, for which the fictional context is relevant and worth mentioning but not overwhelming. The decision of whether a given question along that vein is on- or off-topic is (in a nutshell) "is it about physics or not".

However, I think we need to be significantly more strict when it comes to allowing that type of question on the HNQ list. The reason is simple: upvotes on those threads, on both the question and its answers, are of extremely low quality. The Q&A engine does a (reasonably) good job at vetting the quality of content, both questions and answers, and for regular questions it does an OK job (though with plenty of room for improvement) at vetting the quality of votes. However, for questions about works of fiction, that completely breaks down.

• For the question, upvotes rarely seem to come from a position of "this question is carefully researched, well-posed and well thought through, and it is a useful framework that allows the physics to shine through" and seem to be more along the lines of "this question is about a work of fiction that I like".
• For answers, upvotes have a much stronger bias to favour truthiness rather than actual quality in the ways that it is recognized elsewhere on the site.

Moreover, those problems compound themselves, as the wider-than-usual opening on the upvote tap further reinforces the positive feedback loop that keeps these questions in the list, to the detriment of real physics questions that could be getting that spot on the spotlight. And while the three-day cap definitely helps, the feedback-loop problems remain, so I don't see how this class of questions will behave differently in the currently-live scheme.

Furthermore, the fact that questions like these are so likely to explode in popularity means that they present a very skewed vision of this site's scope on that SE-network-facing window. We get maybe one on-topic question about physics-in-works-of-fiction for every 5-10,000 questions, at best? Then, until the mechanisms are tempered such that the natural rate of these questions appearing on the list is one such question per 10,000 Physics SE questions on the HNQ list, they should be prevented from appearing there.

Answer 3

Let's talk specifics.

And by that, I mean, "let me be the first to shoot my own self-interest in the foot", and argue that this question, which I answered, was rightly kicked off the HNQ list.

What's the deal? Well, the question starts off with the title

Is the Born rule indeed wrong?

and it is basically a request to verify the correctness of an unpublished arXiv eprint. As pointed out in the comments, the consensus view on peer-review questions leans towards classing this question as Too Broad, but I answered it because the literature it was referring to was complex enough for an outsider to puzzle out on their own (in short: a second preprint commented on the paper to point out flaws in the argument, and the authors responded by adding an appendix) but still containable enough that an answer in our format can explain what the deal is. And, also, the paper is awful enough that its flaws can be explained reasonably quickly.

With that in mind, I think that question is on-topic and should remain open, but I do not think it's a good fit for the type of questions we want representing our site on HNQ.

• For one (though I didn't realize it before posting), the title is clickbait, pure and simple. It can be improved, sure, but I'm not sure I see how far up the slope from "terrible" back towards "vaguely reasonable" it can be brought.

  Moreover, the contentiousness of the title (which cannot really be removed, even if it is softened to something more accurate like "Is this paper, which claims the Born rule is incorrect, right?") simply isn't backed up by a particularly well-thought-through question.

• More to the point, keeping it on HNQ is just giving free advertisement to a truly awful paper, which really doesn't deserve it.

Though then again, those are pretty squishy reasons; I wish I could firm it down more but that's the best I've got. (For full disclosure: it got on HNQ and then kicked off before I had time to react - but my first reaction to finding out about the former was that I needed to flag it so it got off.) I'm not sure how much of this can be distilled into a clear policy going forward, and I'm looking forward for people's reaction to this to see how wild off the bat it is, and whether a more clear fingerprint of what made this a bad fit (which can then be used as a benchmark to compare against in future cases) can be hammered out.

Still, though, I think the core point is pretty strong: this was not a question that we want to represent the site, if nothing else because "yeah, we'll fact-check your shallow questions about fifteen-year-old abandoned preprints" isn't really the kind of message I think we should be sending.

Answer 4

This initiative appears to have petered out, but this question is an excellent example of a bad HNQ question, for a wide variety of reasons.

The question was characterized, in answers, comments, and chat, as a "John Rennie vs. safesphere" wrestling match, or even a battle of relativity vs. crackpots, when it was really about a much more boring technical point where both users were technically right. For example, we have this appealing exchange in the comments:

I dont get the answer yes or no?

So there are currently 36 votes for Yes, 0 votes for No (because there was no answer daring to argue that point of view, other than an answer, not by safesphere, that got deleted after receiving -11 votes), and 0 votes for “what does ‘bend spacetime’ even mean?”. Yes has won.

This seems to imply that basic facts of general relativity need defending, even though neither JR nor safesphere said anything in contradiction with them, and that the "winner" is decided by random coders that come over from StackOverflow and can't even write a complete sentence. This does not give a good impression to practicing physicists; it tells them that posting here is a waste of time.

Even more unappealing is that, in the process, many non-mainstream commentators did come out of the woodwork, filling the comments with nonsense. About half were regulars here, and half were random coders. At the same time, I got a pile of downvotes, presumably from users who thought that I was siding "against" John Rennie.

The quality of the discussion is just quite low, and brought lower by the HNQ list, and a good indicator of why we have so many fewer practicing physicists than MathOverflow has practicing mathematicians. It shouldn't deserve so much attention, but instead it's on track to being the most viewed question this week.