25
$\begingroup$

Reputation is a rough measure of a user's experience with the site. It can also be used as a (not very accurate) proxy for their overall expertise in physics.

Our user population consists of people who ask lots of questions and answer few of them, experts who answer lots of questions and typically ask few of them, and people in between. A good measure of a given user's position in this spectrum is the number of questions they've asked over the number of answers they've posted.

Is there a correlation between these two metrics?

$\endgroup$
2
  • $\begingroup$ I just got a silver badge for tenacious which is proof that it's not universally true. To have such a badge defined suggests that there is at least a bimodal distribution. $\endgroup$
    – JDługosz
    Commented Mar 17, 2015 at 18:51
  • $\begingroup$ Related: meta.superuser.com/questions/7707/… $\endgroup$
    – bwDraco
    Commented Mar 21, 2015 at 1:06

2 Answers 2

Reset to default
31
$\begingroup$

Yes, it does.

I tried exploring this using this SEDE query, whose results go something like this:

enter image description here

This is a log-log plot and it shows two distinct regimes, with a definite linear correlation in the higher-rep side. This means that the site's user base really does consist of two different populations with very different behaviours: novice users with reps below ~200 for whom there's no correlation between reputation and question/answer ratio, and more experienced users who ask noticeably less questions at higher reputations.

The dependence in this regime is roughly Q/A = 1/rep, with r squared of about 0.25. I'll leave more detailed statistics to people who can do them right.

One important thing to note is that the query returns results for only 3579 users, which is about 5% of our total of ~72k users, because it only reports on users that have ≥1 answer and ≥1 question. This could be a bug in the SQL, and it's something to keep an eye on. Maybe someone can extend this to cover those cases? It's not clear to me how you'd include those users in the log-log plot below, where they'd form a 'ceiling' at Q/A=∞ of pure askers and a 'floor' at Q/A=0 of pure answerers.

$\endgroup$
11
  • 1
    $\begingroup$ More interesting is why there are those horizontal lines. Also, I'd wager that a lot of people probably signed up to the site just to upvote HNQ here. $\endgroup$
    – Kyle Kanos
    Commented Mar 10, 2015 at 12:59
  • 8
    $\begingroup$ The horizontal lines correspond to ratios of small integers, so I don't really find it that surprising. They're a consequence of the discrete possibilities for question numbers and answers. They taper out after a certain rep as Q/A~2 becomes possible via ratios like 19/10 or 21/10, as opposed to 2/1. $\endgroup$
    – Emilio Pisanty
    Commented Mar 10, 2015 at 13:18
  • 1
    $\begingroup$ Ah, that makes sense now. $\endgroup$
    – Kyle Kanos
    Commented Mar 10, 2015 at 13:23
  • $\begingroup$ Maybe it's just me, but I see a hint of a third population -- in the mid-rep range, there is a branch of the distribution with a positive Q/A-rep correlation. Put another way, if you created some sort of contour on the right-hand side (e.g. the 90th percentile in rep at each Q/A bin), there appears to be a statistically significant minimum in rep at Q/A of ~2. $\endgroup$
    – user10851
    Commented Mar 12, 2015 at 2:49
  • $\begingroup$ @ChrisWhite That definitively sounds plausible, but I don't really have the stats chops to disentangle that. Maybe some of our more statistically minded users could have a look at the data and see what they can find? $\endgroup$
    – Emilio Pisanty
    Commented Mar 12, 2015 at 12:30
  • 3
    $\begingroup$ I am one of the Q=0 people who don't get represented on your otherwise excellent graph. You could use a statistical measure of log(Q/A) to deal with the singularity. A simple approach might be (I did not fully think this through): if you can say "next post is either a question or an answer with probability p", then you find a limit on p given the "asked 0 answered N" data, where $(1-p)^N < 0.5$ (for example). Then you solve the equality for p and plot log(p/(1-p)) instead. $\endgroup$
    – Floris
    Commented Mar 12, 2015 at 15:48
  • 2
    $\begingroup$ What accounts for the rep disadvantage Questioners are at by only getting +5? $\endgroup$
    – Mazura
    Commented Mar 13, 2015 at 7:26
  • $\begingroup$ I just fiddled around a bit with the idea I floated yesterday, and if you use $ln(2)/N$ as your Q/A ratio for the case where $Q=N, A=0$ you will come out to about the right place. Obviously you would use the reciprocal when $Q=0, A=N$ $\endgroup$
    – Floris
    Commented Mar 13, 2015 at 16:55
  • 1
    $\begingroup$ How about just plotting the $(Q+1)/(A+1)$ ratio (or its logarithm)? $\endgroup$
    – hmakholm left over Monica
    Commented Mar 15, 2015 at 16:19
  • $\begingroup$ @ChrisWhite: If you ignore a single outlier (the highest-rep dot with Q/A>10), it begins to look more like the "third population" is just the faint outer end of a symmetric trumpet-shaped "novice" population that overlaps the "experienced" population between roughly 500 and 2000 rep. $\endgroup$
    – hmakholm left over Monica
    Commented Mar 15, 2015 at 16:27
  • $\begingroup$ For Reputation <= 200, there is a positive correlation of 0.087, which according to the standard parametric hypothesis test is highly significant, p = 5e-5. Spearman correlation 0.084 and nonparametric test p = 9e-5. $\endgroup$
    – A. Donda
    Commented Mar 19, 2015 at 18:00
12
$\begingroup$

I think the story is both more complicated and more simple, and it helps to look at questions and answers separately:

The superimposed curves are very strongly smoothed versions of the data, using a cubic smoothing spline (Matlab's csaps with parameter p = 0.1).

According to this, with increasing reputation a user has more and more questions, but at some point (about 2000) this increase levels off towards about 10 questions. The smoothed curve is not a particularly good description of the data though. (I'm ignoring the little dip for reputation < 3, which is driven by users with 1 reputation and several questions. Similar for answers.)

For answers vs reputation, the initial increase is similar, but instead of leveling off it becomes a linear relationship in the log-log plot, which implies a power law behavior. The exponent is about 1.11, which means the increase is only slightly faster than linear.

Comparison of the two smooth curves suggests that increase in Reputation is more driven by questions for small reputation, but more driven by answers for high Reputation, with a crossing point at about 370 Reputation and 4.5 both Questions and Answers.

This explains the picture when looking at questions/answers vs reputation. Until a reputation of about 100, the ratio increases slowly up to a value of 1.4, and then decreases faster and faster.

The "third population" speculated about in the comments might appear due to the fact that both the number of Questions and the number of answers vary wildly in the "medium" range of Reputation around 1000. It would be interesting to see whether these relations observed across users also describe the evolution of users over time.

To me this suggests that the number of answers alone is a better match for reputation that the ratio questions/answers, since the smoothed relationship is monotonous starting from a reputation of about 8.5, i.e. across 94% of users.

$\endgroup$
5
  • $\begingroup$ Somehow, "a fool can ask more questions than seven wise men can answer" sprang to mind when I saw your analysis. $\endgroup$
    – Floris
    Commented Mar 21, 2015 at 21:42
  • $\begingroup$ @Floris, I can't figure out whether that's intended as an insult or not. $\endgroup$
    – A. Donda
    Commented Mar 26, 2015 at 16:32
  • $\begingroup$ If you knew me, you would realize that it is never my intention to insult. The point of the quote (where "fool" is used in the sense of the "fool of the court", or court jester) is that there is a never ending source of questions, and relatively few people capable of answering. So that high reputations tend to be built by a handful of people with the answers. I suspect that if you plotted "number of people who have asked at least N questions" vs "people who answered N" as a parametric function of N you would also see something interesting. $\endgroup$
    – Floris
    Commented Mar 26, 2015 at 16:38
  • $\begingroup$ @Floris, thanks for clarifying, your comment was completely opaque to me. Particularly so since I don't really know you, though I believe we have crossed paths on SO one or two times. Thanks for the suggestion, I'll have a look. $\endgroup$
    – A. Donda
    Commented Mar 26, 2015 at 17:55
  • 1
    $\begingroup$ No problem - glad you asked so I could clear it up. And yes I believe we have on occasion visited the same post... although these days I spend more time on the physics site and less on SO. Incidentally - the Fool was an extremely important person at Court, since he was the only one that could tell the king the truth with impunity - wrapping his criticism of the king in a joke, getting a laugh from the courtiers, and keeping his head. And a smart king would actually pay attention... $\endgroup$
    – Floris
    Commented Mar 26, 2015 at 18:27

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .