HT to Danu on this one.
So there was this recent post inquiring about answering 0-answer questions in which a comment read,
It is not evident that votes on answers to old questions are fewer than on new ones
and, elsewhere on the site, it was mentioned that late answers are typically either really amazing or really bad.
So are either of these statements true? Do older questions get fewer votes? Is there a bi-modality of score in late answers?
Using my recent Data Query Age-Score correlation, I gathered the average net scores of answers as a function of the number of days between the answer posting date and the question posting date. The figure below1 shows the result of the query:
There is a slope at the early part (t ~< 200), suggesting that the most upvotes come when the answer is posted nearest to when the question is posted (which makes sense, especially when thinking about the Hot Network Question Effect). There is also a relatively flat net positive score for t >~ 200, with obvious outliers all over the place2.
Zooming in to the -5 < y < 5 range, we see a rather vacuous region that I can only presume is populated by those who have deleted their answer after a few negative votes early on3, while the bottom right is populated by those who've likely left one crack-pot answer & never came back.
Referring back to either figure, we can see the long "stripes" appearing in the data. This is due to the fact that there are fewer answers that are also down/upvoted less than early on. So this appears to me that there are indeed fewer votes for older answers.
With a 300 day delay between Q & A, there are a grand total of 633 answers with a net positive score, 158 with a net negative score and 498 answers with zero score. Those nonzero scores drop to 385 answers with >= 1 and 77 with <= -1, leaving 329 mostly okay answers (that is, most of those remaining have a score between 0 and 1)--that's an 39.2% and 51.3% decrease in answer count, respectively.
It seems that slightly more than half the Late Answers have a score > |1|. Those nonzero score counts decrease to 143 net positive and 26 net negative when using |Score|>2 as the metric (a 77% and 83% reduction). So it appears to me that there is no clear bimodality, based on the numbers4
Taking the log of the delay between Q & A, we can see a slightly different picture (also using the -5 < y < +5 range):
This seems to suggest the same thing as above: there are lots of "just okay" answers, few really bad ones, and a fair number of really good ones. But since bad ones tend to get deleted, it's kinda hard (for me at least) to completely rule out a complete bimodality, so I'll say this one is plausible.
1 I've posted my (really simple) Python script for plotting this on my
github page
2 The eye-catching (311,20) is entirely due to this post by Ron Maimon. You can easily (fork and) modify this Data Query to catch for whatever AnswerScore=N or age you want--be warned though, this can fail due to sheer volume of results.
3 Though it seems strange that there are so few values close to 0 net upvotes in the t < 200 range
4 Admittedly, we have some heavily downvoted Late Answers that are deleted and thus cannot be accounted for in this mini-study.
This seemed like an interesting question, so I figured I'd do some research, too. I started with Kyle's SEDE query, and made a few changes to it:
Since the original question was about answering old questions with no answers yet, I figured it would be more meaningful to only look at the first answer to each question.
Kyle's query bins the ages by day, but doesn't show how many answers go into each bin. That was easy enough to add, so I did.
As it turns out, the age distribution is heavily skewed to the left, with almost all first answers being posted during the first day (and thus falling into a single bin in Kyle's plot), while the higher bins often had just one answer in them, and so ended up with a lot of noise. To make the binning more reasonable, I switched to a logarithmic age scale, taking the base-10 logarithm of the difference between question and answer ages, and binning it at intervals of 0.1.
Finally, to check how much variation I was averaging over, I also added the sample standard deviation for each bin into the output.
The modified query is here. Saving the output as CSV and playing with it in Excel produced the following bubble plot:
The area of each bubble is proportional to the number of answers in the bin. The dotted trend line is autogenerated by Excel, and I suspect it doesn't actually account for the weight in each bin, but even just eyeballing the bubbles does confirm a clear downward trend (after the first 5 minutes or so, anyway).
In any case, the unweighted trend line shown above is perhaps more relevant anyway, if we're mostly interested in answers to old questions (i.e. the right-hand side of the plot); a weighted trend line would be dominated by the age range between 5 minutes and 1 day, during which most first answers are posted, and would probably show an even stronger downward trend.
In any case, it's worth keeping in mind that these averages conceal a lot of noise. To show how much, here's the same plot with ±1 s.d. error bars:
And keep in mind that those are just 1σ error bars, meaning that (assuming a roughly normal score distribution, which may or may not be valid), almost one third of all first answers lie even further away from the mean.
Anyway, I would say that, qualitatively, these results confirm Kyle's general conclusion: there is a downward trend in answer score as a function of question age. That said, clearly there's also a lot of variation not explained by age alone — good answers still get more upvotes than bad ones, regardless of how old the question is.
Ps. Just for comparison, here's a combined plot showing the same query results for Physics.SE, Math.SE and Stack Overflow:
The bubble sizes for SO have been scaled down by a factor of 10 compared to the other sites; SO is just that big.
Also, especially for SO, there are some funny outlier data points that fall of the left and/or top edge of the plot, but the sizes of those bins are so small that they may be mostly just noise. Still, it's kind of curious to note that there are, apparently, 82 first answers on SO posted (almost) exactly 10 seconds after the question, with an average score of 49.73(!).
Pps. It seems that the huge average scores, mentioned above, for first answers posted just a few seconds after the question are mostly due to self-answers. Apparently, before SE added the feature to allow you to post a question and an answer at the same time, what some people used to do was to write the answer in advance, save a copy locally, write and post their question and then immediately answer it afterwards. Clearly, these answers were often excellent, and thus gained a lot of upvotes.
I went and updated the SEDE query to exclude self-answers and re-ran it, but it turns out that, above one minute (≈ 101.8 seconds) or so, the differences are minimal. Evidently, the sharp downward trend between 1 and 5 minutes, especially on SO, is due to other causes. Anyway, the last plot above now excludes self-answers.
Ppps. I compared the plots more carefully, and noticed that excluding self-answers raises the average first answer score in the 105 seconds (≈ 1 day) to 107 seconds (≈ 3 months) range noticeably (by about 0.1 to 0.2 points or so) on Math.SE and SO. Apparently, there's a non-negligible fraction of users on those sites who, if they don't get any answer to their question, will eventually post a self-answer that is often poorly received. The effect here on Physics.SE is less clear, but that may be simply due to the smaller number of answers here.
