15
$\begingroup$

Earlier today I was eyeballing other people's badges because, you know, badges, and I came upon something of a curious observation. If you look at the answer badges - specifically Nice, Good and Great Answer, there is a clear sense that, even for the bronze badge, established users of the site - and particularly users with, say, 1k+ rep - get a lot of the badges. However, if you look at the users who have earned the gold question badges, the picture is completely different: Famous Question, Great Question and Stellar Question have been earned overwhelmingly by <1k rep users; ditto for the silvers and bronzes.

This is of course not news. This was discussed in depth in Question self-destruction: why don't experts ask more questions?. It is also not a surprise in any sense that more questions come from lower-rep users and more answers come from higher-rep users - that's to be expected in some form in any Q&A site.

However, I'm quite struck at how we do have a pretty big set of really nice questions, and that they were mostly asked by quite low-rep users. While it is natural that questions will tend to be asked by lower-rep users, it is not a given that the good questions will come from that end of the scale.

I therefore decided to explore this theme a bit and, in the meantime, dredge up what's left of my SQL skills after ten years (!) of rust. I came up with this query for the Stack Exchange Data Explorer: What reputation do people with Popular/Notable/Famous Question badges typically have?. It produces a cumulative-distribution-function-like plot of the number of users below reputation rep that have asked a question with >viewsThreshold views. For the gold badge, the plot is this:

enter image description here

Over 90% of such questions were asked by users who (even now) have less than 1k. There are a few between 1k and 2k, and really not much other than this great (cw) question of Mark Eichenlaub and this one of nibot's. I genuinely think this is surprising and really puts the nail on what we knew all along: 'expert' users need to start asking more questions.

It also raises an interesting question as regards why this happens: can it really simply be explained by low-rep users "stumbling" on the good nuggets of question space, as part of the huge body of questions asked by the overwhelming majority of users that have <1k rep? Looking at, say, the questions that earned the Good sobriquet, I think there is something else at stake. Those questions have a simplicity, and therefore a generality, that I think tends to be lacking in the questions asked by more established users. What do people think about this?

For comparison, I have also cooked up the corresponding query for the Nice/Good/Great Answer badge series, What reputation do people with Nice/Good/Great Answer badges typically have? . There aren't enough Great Answers to give a smooth enough plot, but the silver earners, at 25+ votes, are distributed like this:

enter image description here

Note: (1) the horizontal scale is very different, (2) the slope is much shallower, and (3) only 20% of silver-earning answers come from <1k users. (And, also, multiple good answers from Luboš M, John R, Anna V, David Z, and Mark E.)

Finally, for completeness, here are the corresponding queries for the other two question badge series: What reputation do people with Nice/Good/Great Question badges typically have? and What reputation do people with Favorite/Stellar Question badges typically have?. Both have, as far as I can tell, the same behaviour as the views-based series.

I'm well aware, of course, that so far I've been talking a lot but there aren't many actual questions. The main motivation of this post is to share this observation and the (moderately hard and solid) statistics that back it up. I guess the real question is something like: what do people make of this? Does this correspond to the image this community has of the site? Or is there something missing from this analysis. What are the core reasons for this to happen? Is it simply a volume thing - i.e. the bulk of questions tends to come from lower-rep users than higher-rep ones - or is there indeed some core simplicity and accessibility to those great questions that eludes many of our great answerers? (I think there's quite a bit to that. And, in that case, we should really go and learn how to ask from those great questions.) If this is happening, is it a good thing? Is it a neutral, that's-how-things-are, thing? Or is it something we should try to counter-act? If so, how?

In response to the comments and the Pulsar's and Manishearth's answers I'd like to say two things.

One is that of course question score is only an approximate, stand-in metric to the completely undefinable "question quality". However, I feel it is not entirely a bad metric, particularly at the high end of the scale: most of the badge-earning questions are indeed good questions. (That's not to say, on the other hand, that all our good questions get high score, and particularly so for high-end questions.) But there's nothing wrong with a highly-voted, low-level question: if it gets good physics across to a lot of people, at a level where they can understand it, what else could you ask?

I'm on this site, mostly, to refine my outreach-fu, and I suspect most high-end users are here at least partly for that. As I see it, these questions are a very valuable resource. Seriously: go read our Good Answers and tell me you didn't have fun. And, at least, it's something we can and should learn from.

$\endgroup$
13
  • 3
    $\begingroup$ I will say that having high-rep users asking questions is probably a good sign of an expert-level site. $\endgroup$
    – David Z
    Dec 8, 2013 at 7:12
  • 2
    $\begingroup$ That's the thing. High-rep users are by and large not asking a lot of questions, and they're definitely not asking the really good questions. (That first assertion is unfortunately purely phenomenological and I can't back it up with data. But I think these queries make a strong case for the second one.) $\endgroup$
    – Emilio Pisanty
    Dec 8, 2013 at 7:16
  • $\begingroup$ I'm sure SEDE could support a query showing the number of questions asked per user per unit time in different rep ranges, and thus we could get data on the number of questions asked by users in different rep brackets. But I think we already know there's a chicken-and-egg problem of sorts, in that high-rep users don't ask their questions because of a perception (often, but not always, correct) that the expertise to answer them isn't here. $\endgroup$
    – David Z
    Dec 8, 2013 at 7:22
  • 2
    $\begingroup$ Note that it's sometimes the really low-level questions which get a lot of attention. $\endgroup$
    – Abhimanyu Pallavi Sudhir
    Dec 8, 2013 at 8:17
  • $\begingroup$ I'm proud of my 2 Good Answers, but am amazed that there's some with 20+ Good Answers... $\endgroup$
    – Kyle Kanos
    Dec 9, 2013 at 3:56
  • 3
    $\begingroup$ I would be interested in seeing the number of good questions as a function of time. I get the impression that the frequency has fallen with time, possibly because most of the obvious good questions have already been asked. $\endgroup$
    – John Rennie
    Dec 9, 2013 at 11:30
  • 5
    $\begingroup$ Main mistake of this website is that it gives you a feeling that low rep users are NOT experts. TOTALLY not true :) I would say even opposite - most of the REAL world experts should be low rep users since they have NO time or DESIRE to hang out here. Last but not least -- expertise has nothing to do with CREATIVITY. Often its exactly opposite to expertise. So if you ARE an expert -- there are high chances you have low creativity in your field. $\endgroup$
    – Asphir Dom
    Dec 9, 2013 at 11:46
  • 1
    $\begingroup$ @JohnRennie That data is probably available in the data explorer. However, older questions naturally have (had more time to get) more upvotes, so I would take any such results with a large grain of salt. $\endgroup$
    – Emilio Pisanty
    Dec 9, 2013 at 12:48
  • 1
    $\begingroup$ From what I have seen, "the experts" don't ask questions at all. I would love to see the statistics of that: from eyeballing I have noticed that users with ~10k rep in Physics, have asked ~10 questions and answered ~100, perhaps even more. Where is the curiosity people? $\endgroup$
    – Physics_maths
    Dec 10, 2013 at 13:56
  • $\begingroup$ @LoveLearning That has been discussed, in quite a bit of depth in the meta question I linked to. $\endgroup$
    – Emilio Pisanty
    Dec 10, 2013 at 13:59
  • $\begingroup$ Aha OK the for the link. $\endgroup$
    – Physics_maths
    Dec 10, 2013 at 14:14
  • $\begingroup$ @EmilioPisanty If you mind to call theory a "data", your statement is known as bike shedding $\endgroup$
    – Val
    Dec 20, 2013 at 12:56
  • $\begingroup$ Logarithmic vertical scales anyone? :D $\endgroup$
    – DanielSank
    Dec 24, 2014 at 4:46

5 Answers 5

Reset to default
19
$\begingroup$

Nice graphs!

I personally thought this would be obvious though. If we use votes as a way of measuring "best" (far from accurate), then the posts with the most votes will be the ones that more people can understand and appreciate. Popular posts cater to this perfectly, and thus get the most votes.

Really good higher level questions are (a) hard to find as more experienced people ask less questions, and (b) understood by less prospective voters. So there will be very few high level posts with a lot of votes.

I wish we had a better way of gauging post quality than votes, especially for higher level posts.

$\endgroup$
1
  • 11
    $\begingroup$ I think this is the main reason questions by low-rep users are more likely to get a high vote count. A great question by an expert user will be understood only by other expert users (and possibly even just a subset of those) which automatically greatly limits the possible vote count. $\endgroup$
    – Wouter
    Dec 9, 2013 at 15:04
19
$\begingroup$

Do not neglect the degree to which the "level" of a question determines the audience that will be interested. There are more laymen than beginners that journymen than masters, and than means that there are more people to be impressed by basic questions than by esoteric, expert level questions.

The result is that it is the really good basic questions and the really good answers to basic questions that get the largest number of votes.

I can see this in the answers that I have given, both here and on Stack Overflow. My highest voted posts are pretty decent answers to fairly basic questions. I've written a few answers on both sites that I consider to be good, expert level work, but they are not the one that the crowd loves. Simply because the crowd that can appreciate them is much small than the one that can appreciate the basic coolness of bootstrapping a complex tool or the idea of huge-scale, slow-motion acoustics in a near-vacuum.

In short: the thing that votes measure is a convolution of the "quality" of a post with the size of the audience that can appreciate it.

$\endgroup$
5
  • 2
    $\begingroup$ Which makes me think the data could be "improved" by conditioning the votes with the number of views somehow. The number of views is really the only metric we have to measure audience size. $\endgroup$
    – tpg2114
    Dec 8, 2013 at 20:27
  • $\begingroup$ @tpg But if a post is of a lower level, more people can understand it, so the views will be higher. In a sense, that is a good post but it excludes niche topics and to a degree high level physics. $\endgroup$
    – Manishearth
    Dec 9, 2013 at 1:52
  • 2
    $\begingroup$ @Manishearth Exactly -- so high votes with high views needs to be weighted such that few votes and few views for niche topics are "equal." Like votes^n/views^p or something that somehow recognizes that awesome questions might not be seen much due to the specificity. $\endgroup$
    – tpg2114
    Dec 9, 2013 at 3:24
  • $\begingroup$ @tpg2114 Oh facepalm I looked at it the opposite way $\endgroup$
    – Manishearth
    Dec 9, 2013 at 4:26
  • 3
    $\begingroup$ @tpg2114 I think fundamentally our problem is trying to project a 2D parameter space (extrinsic breadth, approximated by views, and intrinsic awesomeness, which should be something like votes/views) onto a single scalar quality. Any way we do so loses information. $\endgroup$
    – user10851
    Dec 9, 2013 at 19:55
16
$\begingroup$

It's important to note that the number of upvotes on a question or an answer are not necessarily a sign of quality, but rather of popularity. You could write a great question or answer, but if nobody pays attention to them, they won't get upvoted.

I noticed myself that some of my most-upvoted answers took me the least effort to write, and conversely some answers that took me lots of time got barely any attention at all.

$\endgroup$
2
  • 7
    $\begingroup$ 1 rep users are likely to have almost no Physics knowledge, so they'll ask questions that a broader audience will be able to understand. $\endgroup$
    – jinawee
    Dec 8, 2013 at 15:51
  • 5
    $\begingroup$ This! My two Good Answers were popular questions, which is why it got any attention. I've had A's that I thought were spectacular and didn't even get the Accepted mark. $\endgroup$
    – Kyle Kanos
    Dec 9, 2013 at 4:01
8
$\begingroup$

Training and expertise change how your neural circuitry processes data about the world, causing your interpretation of much of the data about the world to become so rapid and automatic that it can become very difficult even to see why certain questions are even meaningful, let alone "interesting."

This is not just a training issue. The ability to discard many forms of data very quickly appears to be innate in how complex biological neural systems manage to process what would otherwise be a fatal overloads of sensory data -- and I do mean fatal, e.g. via driving a car without such data reductions in effect. This is not just some sloppy form of ignoring data at random, either. Brains in general have an absolutely amazing ability to identify and discard unnecessary data before it gums up the limited processing available, allowing them to focus (foveate) quickly on the much smaller subset of clues and data that is truly critical for survival.

All of this comes into play in understanding why experts ask fewer questions because higher-level intelligence -- sentience -- seems to use these same mechanisms to create largely sensory-isolated loops that can revisit data many times without requiring the data to play again in the real world. Your internal "thought voice" is perhaps the most extreme example, since it is independent of any real external voices. (The bumper sticker version of that might be "I hallucinate, therefore I am.") Insight becomes a matter of applying smart data reduction not to what you are seeing, but to what you are thinking.

The overall result is an abstract view of the world that can become highly disconnected from the reality shouted at us by our senses, guided by mechanisms that become so deeply reflexive that we barely even see other options, let alone question them. That's a good thing in cases such as science or engineering, for which first-order sensory reality often turns out to be a remarkably poor indicator of what is really going on. It's a bad thing when it enables people to abstract away reality in a way that disconnects them from the very data to which they should be paying the most attention.

Do I really need to give examples of the latter?

The goal of science -- what science means at the biological level -- is to balance our ability to abstract data rapidly ("expertise") with the realization that such abstractions are always dangerous, even when applied even slightly beyond the original data. We must never confuse the brain-generated emotion of confidence with experiment-based confirmation from the physical world that those abstractions are still valid.

Now, given all of that, it's time to go back to your fascinating data and charts showing a strong inverse relationship between how many interesting questions get asked and level of expertise of those asking.

Because experts develop and frequently use complex abstraction structures for interpreting questions and data quickly and decisively, those abstraction structures inevitably begin to make heavy use of the fast-discard capabilities of complex biological neural systems. For that very reason, it also becomes more and more difficult without special training for them even to "see" questions or data that fall within the scope of expertise. Thus they are less inclined to dwell on them much or ask many questions about them.

Conversely, relative novices have no such rapid-discard structures in space, so even small discrepancies or obscurities in the overall structure tend to jump out at them, sometimes very starkly. The concept of "present" is a straightforward example. To many relativists and quantum field theorists, the idea of "block time" -- that is, that that the universe preexists from start to finish as a bundle of worldlines -- seems almost inevitable, for reasons too numerable to mention here. Yet to almost anyone not trained in those reasons, the idea of block time often sounds either insane, a flat-out contraction of everything that a person sees in their own lives, or like some form of extreme religious belief. As a result, the questions they will ask about such an issue will be very different from those that an expert might ask, and will also have very different emotional weighting.

There's not easy resolution to all of this, since we are very much products of our own brains. However, with that said, it is nonetheless emphatically possible and very much desirable to train ourselves to be aware of our own limitation, and how to minimize their impacts in our overall analytical approach to understanding the universe. The colloquial phase for it is humility, but it's a form of humility that needs to be sharpened and structured as finely as the flowing patterns of a Damascus sword.

$\endgroup$
3
  • 1
    $\begingroup$ I loved the word "foveate". Now, don't use it too much, though, lest it should be taken over by some self proclaimed management guru hawking their "services" on the corporate world and sucked dry of its meaning by a million middle managers seeking to impress at weekend "training" courses. $\endgroup$
    – Selene Routley
    Dec 12, 2013 at 9:03
  • 1
    $\begingroup$ Heh! It's directly from some ongoing research efforts I'm involved with, but your point about potential abuse of it is scarily plausible! BTW, a long, long time ago, I was giving a talk on how to make software modules easier to reuse later, as opposed to reacting to whatever you had. The word that I needed to describe this preparation-in-advance approach did not seem to exist at the time, so I invented one that captured the opposite-of-reactive thought nicely: "proactive". It seemed harmless enough at the time... $\endgroup$
    – Terry Bollinger
    Dec 12, 2013 at 13:47
  • $\begingroup$ "colloquial phase" -> "colloquial phrase" $\endgroup$
    – ErikE
    Oct 28, 2015 at 16:28
1
$\begingroup$

If you observe the top/highest voted questions (https://physics.stackexchange.com/questions?sort=votes, Cooling a cup of coffee with help of a spoon, A mirror flips left and right, but not up and down, Why don't metals bond when touched together?)

I believe it is safe to presume that these questions are not those which would be poupular just amongst layman but also among physicists. But at the same time we can say that experts would already be knowing such things. As the site deals with existing physics and related questions, i believe that the experts have a good grasp on it and have learnt most stuff. At the same time when beginners ask questions which experts woupd already be knowing they get high votes and hence have these sair badges.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .