9
$\begingroup$

HT to Danu on this one.

So there was this recent post inquiring about answering 0-answer questions in which a comment read,

It is not evident that votes on answers to old questions are fewer than on new ones

and, elsewhere on the site, it was mentioned that late answers are typically either really amazing or really bad.

So are either of these statements true? Do older questions get fewer votes? Is there a bi-modality of score in late answers?

$\endgroup$
14
$\begingroup$

Are there fewer votes on old answers?

Using my recent Data Query Age-Score correlation, I gathered the average net scores of answers as a function of the number of days between the answer posting date and the question posting date. The figure below1 shows the result of the query:

enter image description here

There is a slope at the early part (t ~< 200), suggesting that the most upvotes come when the answer is posted nearest to when the question is posted (which makes sense, especially when thinking about the Hot Network Question Effect). There is also a relatively flat net positive score for t >~ 200, with obvious outliers all over the place2.

Zooming in to the -5 < y < 5 range, we see a rather vacuous region that I can only presume is populated by those who have deleted their answer after a few negative votes early on3, while the bottom right is populated by those who've likely left one crack-pot answer & never came back.

enter image description here

Referring back to either figure, we can see the long "stripes" appearing in the data. This is due to the fact that there are fewer answers that are also down/upvoted less than early on. So this appears to me that there are indeed fewer votes for older answers.

Are the Late Answers either really bad or really good?

With a 300 day delay between Q & A, there are a grand total of 633 answers with a net positive score, 158 with a net negative score and 498 answers with zero score. Those nonzero scores drop to 385 answers with >= 1 and 77 with <= -1, leaving 329 mostly okay answers (that is, most of those remaining have a score between 0 and 1)--that's an 39.2% and 51.3% decrease in answer count, respectively.

It seems that slightly more than half the Late Answers have a score > |1|. Those nonzero score counts decrease to 143 net positive and 26 net negative when using |Score|>2 as the metric (a 77% and 83% reduction). So it appears to me that there is no clear bimodality, based on the numbers4

Taking the log of the delay between Q & A, we can see a slightly different picture (also using the -5 < y < +5 range):

enter image description here

This seems to suggest the same thing as above: there are lots of "just okay" answers, few really bad ones, and a fair number of really good ones. But since bad ones tend to get deleted, it's kinda hard (for me at least) to completely rule out a complete bimodality, so I'll say this one is plausible.



1 I've posted my (really simple) Python script for plotting this on my github page
2 The eye-catching (311,20) is entirely due to this post by Ron Maimon. You can easily (fork and) modify this Data Query to catch for whatever AnswerScore=N or age you want--be warned though, this can fail due to sheer volume of results.
3 Though it seems strange that there are so few values close to 0 net upvotes in the t < 200 range
4 Admittedly, we have some heavily downvoted Late Answers that are deleted and thus cannot be accounted for in this mini-study.

$\endgroup$
  • $\begingroup$ Dang, that outlier may be the longest post I've seen on any SE site ever. $\endgroup$ – tpg2114 Mar 29 '15 at 3:21
  • 1
    $\begingroup$ Then read/look at Lubos's answer $\endgroup$ – Kyle Kanos Mar 29 '15 at 3:29
  • $\begingroup$ We should start a "novels of SE" coffee table book. That's crazy long. $\endgroup$ – tpg2114 Mar 29 '15 at 3:31
  • 2
    $\begingroup$ Wow! This is a lot of scientifically (statistically) oriented effort in ascertaining the validity of a comment. $\endgroup$ – 299792458 Mar 29 '15 at 7:10
  • $\begingroup$ @KyleKanos Answer? Don't you mean essay? $\endgroup$ – JamalS Mar 29 '15 at 16:16
  • $\begingroup$ Nice analysis! But how can the outlier be Ron Maimon's answer if the vertical axis is supposed to be average score? Unless it were the only answer given 311 days after the questions was posed, which it doesn't seem to be. $\endgroup$ – Wouter Apr 2 '15 at 20:21
  • $\begingroup$ @Wouter: Aha, good catch. It wasn't the post I linked, it actually was a different one with ~160 votes on it. There were 8 total questions posted on date difference 311, that one sets them apart. $\endgroup$ – Kyle Kanos Apr 2 '15 at 21:29
  • $\begingroup$ @KyleKanos Ah yes, that ought to do it :) $\endgroup$ – Wouter Apr 2 '15 at 23:45
11
$\begingroup$

This seemed like an interesting question, so I figured I'd do some research, too. I started with Kyle's SEDE query, and made a few changes to it:

  • Since the original question was about answering old questions with no answers yet, I figured it would be more meaningful to only look at the first answer to each question.

  • Kyle's query bins the ages by day, but doesn't show how many answers go into each bin. That was easy enough to add, so I did.

  • As it turns out, the age distribution is heavily skewed to the left, with almost all first answers being posted during the first day (and thus falling into a single bin in Kyle's plot), while the higher bins often had just one answer in them, and so ended up with a lot of noise. To make the binning more reasonable, I switched to a logarithmic age scale, taking the base-10 logarithm of the difference between question and answer ages, and binning it at intervals of 0.1.

  • Finally, to check how much variation I was averaging over, I also added the sample standard deviation for each bin into the output.

The modified query is here. Saving the output as CSV and playing with it in Excel produced the following bubble plot:

Plot of average first answer score vs. time since the question was asked on Physics Stack Exchange

The area of each bubble is proportional to the number of answers in the bin. The dotted trend line is autogenerated by Excel, and I suspect it doesn't actually account for the weight in each bin, but even just eyeballing the bubbles does confirm a clear downward trend (after the first 5 minutes or so, anyway).

In any case, the unweighted trend line shown above is perhaps more relevant anyway, if we're mostly interested in answers to old questions (i.e. the right-hand side of the plot); a weighted trend line would be dominated by the age range between 5 minutes and 1 day, during which most first answers are posted, and would probably show an even stronger downward trend.

In any case, it's worth keeping in mind that these averages conceal a lot of noise. To show how much, here's the same plot with ±1 s.d. error bars:

Same plot as above, with one sigma error bars

And keep in mind that those are just 1σ error bars, meaning that (assuming a roughly normal score distribution, which may or may not be valid), almost one third of all first answers lie even further away from the mean.

Anyway, I would say that, qualitatively, these results confirm Kyle's general conclusion: there is a downward trend in answer score as a function of question age. That said, clearly there's also a lot of variation not explained by age alone — good answers still get more upvotes than bad ones, regardless of how old the question is.


Ps. Just for comparison, here's a combined plot showing the same query results for Physics.SE, Math.SE and Stack Overflow:

Average first answer score vs. logarithmic question age for Physics.SE, Math.SE and SO

The bubble sizes for SO have been scaled down by a factor of 10 compared to the other sites; SO is just that big.

Also, especially for SO, there are some funny outlier data points that fall of the left and/or top edge of the plot, but the sizes of those bins are so small that they may be mostly just noise. Still, it's kind of curious to note that there are, apparently, 82 first answers on SO posted (almost) exactly 10 seconds after the question, with an average score of 49.73(!).

Pps. It seems that the huge average scores, mentioned above, for first answers posted just a few seconds after the question are mostly due to self-answers. Apparently, before SE added the feature to allow you to post a question and an answer at the same time, what some people used to do was to write the answer in advance, save a copy locally, write and post their question and then immediately answer it afterwards. Clearly, these answers were often excellent, and thus gained a lot of upvotes.

I went and updated the SEDE query to exclude self-answers and re-ran it, but it turns out that, above one minute (≈ 101.8 seconds) or so, the differences are minimal. Evidently, the sharp downward trend between 1 and 5 minutes, especially on SO, is due to other causes. Anyway, the last plot above now excludes self-answers.

Ppps. I compared the plots more carefully, and noticed that excluding self-answers raises the average first answer score in the 105 seconds (≈ 1 day) to 107 seconds (≈ 3 months) range noticeably (by about 0.1 to 0.2 points or so) on Math.SE and SO. Apparently, there's a non-negligible fraction of users on those sites who, if they don't get any answer to their question, will eventually post a self-answer that is often poorly received. The effect here on Physics.SE is less clear, but that may be simply due to the smaller number of answers here.

$\endgroup$
  • 1
    $\begingroup$ SO is just that crazy. $\endgroup$ – Emilio Pisanty Apr 1 '15 at 18:12
  • $\begingroup$ (That said, is there some quick way to query for those answers?) $\endgroup$ – Emilio Pisanty Apr 1 '15 at 18:13
  • 1
    $\begingroup$ I've been involved in those answer races on SO before. It usually goes something like -- See the question and immediately post a single sentence answer. Edit in another sentence. Then edit in another. And so on. Until it becomes a full answer. Voting over there seems to depend on who showed up first since they have enough users that each question will have 5-10 people answering in nearly the same time frame. $\endgroup$ – tpg2114 Apr 1 '15 at 18:43
  • $\begingroup$ Yes, there is now. I didn't find the results that interesting, though. $\endgroup$ – Emilio Pisanty Apr 1 '15 at 19:01
  • $\begingroup$ @EmilioPisanty: Seems that the weird sub-minute outliers on SO are mostly self-answers. Excluding those, as you did in your query, makes the average scores a lot more sensible (and also heavily reduces the total number of such ultra-fast answers). The steep downward slope between 1 and 5 minutes is still there, though. $\endgroup$ – Ilmari Karonen Apr 1 '15 at 20:33
  • $\begingroup$ Huh. Well, something came of it. As an aside, could you add vertical lines on your plots to show where 1 minute / 1 hour / 1 day / 1 month / 1 year are? I find the horizontal axes very unintuitive. $\endgroup$ – Emilio Pisanty Apr 2 '15 at 13:11
  • $\begingroup$ Gotta love data analysis :) $\endgroup$ – Wouter Apr 2 '15 at 20:27
  • $\begingroup$ @Emilio: I guess I could, at least by editing the images in GIMP if by no other means. In the mean time, here's a quick cheat sheet: 1 sec = 0.0, 1 min ≈ 1.8, 5 mins ≈ 2.5, 1 hour ≈ 3.6, 1 day ≈ 4.9, 1 week ≈ 5.8, 1 month ≈ 6.4, 1 year ≈ 7.5. $\endgroup$ – Ilmari Karonen Apr 8 '15 at 18:35

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .