18
$\begingroup$

There has been a general sense among a number of users, myself included, that simplistic, almost throwaway answers can get very high scores, while thoroughly researched, high-level answers can languish with few to no votes. See the discussion here for example.

To what extent is this backed by data?

I set out to answer this question, if not once and for all then at least to begin to shed some light on it. To that extend I wrote a very minimal Python script, uploaded to GitHub, for crunching some numbers and making some plots.1

The script needs two pieces of input to run. First, it expects a local copy of the CSV file associated with this Data Explorer query. Simply run it with your user ID and click the "Download CSV" link. The other requirement is much more labor intensive: A file with the post IDs of all your answers and a "quality score" for each. For my purposes, I used a 6-point scale:

0. It might actually be beneficial for this answer to be deleted.
1. Just a quick answer to an easy question, one that required essentially no work.
2. A not-so-great answer that was nonetheless not entirely trivial.
3. A run-of-the-mill good answer.
4. A solid, well-researched answer that provides thorough insight into the problem at hand.
5. An outstanding answer, the kind about which songs will be sung many generations hence.

This is really unavoidable for my scheme. In order to measure the correlation between "quality" and score, we need to actually have "quality" numbers on hand. I did my best to rate all my 400 or so answers to date, trying not to let the answer's votes sway my feelings about its "quality" one way or the other. It took the better part of a lazy afternoon to go through them all, in case anyone wants a time estimate for doing this themselves.

Some plots and conclusions are given an an answer below. However, this is just the analysis of my own opinion of my own answers. More data would be great, so I encourage others to use the tools mentioned above (or their own tools) for analyzing this sort of data.2

Please feel free to post your own data or your alternate analysis of my data as answers here.

1Bug reports, feature requests, and support inquiries welcome. I'll try my best, but I cannot guarantee a timely response.

2My personal data dump and self-rankings are included in the repository, in case you want to jump right in and play with data without ranking answers. In particular, the query returns more information than the script currently uses, such as dates and question scores, so a better analysis can certainly be done.

$\endgroup$
12
  • 1
    $\begingroup$ Nice!!!!!!!!!!! $\endgroup$
    – Danu
    Commented Feb 21, 2015 at 12:28
  • 2
    $\begingroup$ Well now I know what to do when I'm bored waiting for a run to finish.... $\endgroup$
    – Kyle Kanos
    Commented Feb 21, 2015 at 16:08
  • 4
    $\begingroup$ Sounds like we need to have an anonymous review board for ranking answers of others :) It would actually be really, really interesting to see, given the same answers, how different users rank the quality of them. That alone could make for some really interesting analysis. $\endgroup$
    – tpg2114
    Commented Feb 21, 2015 at 17:33
  • $\begingroup$ I like to think many generations of people to come will sing songs about all of my answers $\endgroup$
    – Jim
    Commented Feb 23, 2015 at 14:59
  • $\begingroup$ @tpg2114 I was thinking the same; it'd be pretty neat to have an external website that shows you an answer without its score and asks you to rate it. $\endgroup$
    – David Z Mod
    Commented Feb 23, 2015 at 23:54
  • $\begingroup$ @DavidZ Could be fun to turn it into a game -- "Guess the score of the answer." Make it so if you change focus from the window, it moves on to the next question without ranking the one shown so you can't just search for the question on SE. If somebody wanted to cheat so bad they had two devices to look it up, whatever. It's just for fun! $\endgroup$
    – tpg2114
    Commented Feb 24, 2015 at 0:05
  • $\begingroup$ An alternative rule could be to use the number ranking given in this question (0-5) and once a certain number of votes are cast, you get points based on how far from the average you are. So if you give something a 0 and X people give it a 5, you wouldn't get any points. $\endgroup$
    – tpg2114
    Commented Feb 24, 2015 at 0:06
  • $\begingroup$ @tpg2114 not to say that wouldn't be interesting in its own right, but for what I had in mind the point is explicitly to give the answer a quality rating that is not related to (or, not influenced by) its vote score. $\endgroup$
    – David Z Mod
    Commented Feb 24, 2015 at 1:32
  • 1
    $\begingroup$ @DavidZ I was going to write a script to do something like that, but I don't trust my high-level network programing. I'm always afraid I'll accidentally launch a DoS attack on a website. $\endgroup$
    – user10851
    Commented Feb 24, 2015 at 1:34
  • $\begingroup$ @DavidZ Nobody would need to know how close/far they were from the actual vote counts. They would read the answer, "guess" the score and/or assign the 0-5 rank to it, and never know the answer. They would just know at the end of the day/week/whatever how good they are at evaluating the questions relative to other people evaluating questions. Basically turn our desire to get data into a game so there's actually motivation to play along with us $\endgroup$
    – tpg2114
    Commented Feb 24, 2015 at 1:41
  • $\begingroup$ The real issue might be such a site would run afoul of the CC-BY-SA license if it stripped author's identification and a source link... Not that I think SE would care for something like this though. $\endgroup$
    – tpg2114
    Commented Feb 24, 2015 at 1:46
  • 1
    $\begingroup$ @tpg2114 well, yes, but it's good to abide by the license just because it's the right thing to do. Anyway, one can still hide the post score. This would be an interesting thing to discuss in chat. $\endgroup$
    – David Z Mod
    Commented Feb 24, 2015 at 2:16

2 Answers 2

Reset to default
16
$\begingroup$

One thing that can be done is to group answers by quality and look at the distribution of answer scores within each group. Below is a plot of the means (circles) and 25-75 percentiles (error bars) for the six quality groups mentioned above. From quality 0 to quality 5, the numbers of answers I put into each category are 43, 79, 80, 137, 46, and 19.

trend

Somewhat surprisingly, there does seem to be a positive correlation: "better" answers get more upvotes. It's not a particularly strong correlation compared to how wildly my answers vary in quality, but it's there. Note I haven't done any sophisticated fits to this data, this is all just qualitative visualization.

In order to better capture the underlying distributions being summarized above, I also plotted histograms of answer scores for each group, shown below.

histogram

Some explanation is warranted. I chose to use logarithmic abscissas.1 If there were N answers with a score of A, I took this to mean an amount N was distributed uniformly on (A-1/2, A+1/2). This distribution was then split up into the appropriate logarithmic bins based on how much they intersected the interval. Anything with a negative score was put into the separate bar on the left. Anything that would have been put into a bin below my cutoff (10^0) is also put in the separate bar, including all scores of 0 (since (-1/2, 1/2) lies below 10^0) and some scores of 1 (since part of (1/2, 3/2) lies below 10^0).

Here I reach a similar conclusion: quality is not negatively correlated with score, and in fact it's a little bit positively correlated. Sure, 1 out of 19 of my personal "best" answers has been sitting for months with a score of 1, but a couple of them have done somewhat well. And a lot of my in-my-estimation-worthless answers have received the scores they deserved.

It's also apparent that at least my answer score distributions have awkward long, low tails to the right. This is probably a signature of a small fraction of answers going viral and getting lots of votes. We tend to focus on the low quality answers that get scattered into the very high bins, but this happens at all quality levels. In fact, the distributions truncated above 10^1 have a nice quality-score correlation.

1Perhaps because I'm an astronomer, and we love using unintuitive axes for all our plots. Just be grateful I didn't make the x-axis run from right to left.

$\endgroup$
4
  • 3
    $\begingroup$ Great: Goes to show how we all suffer from confirmation bias! $\endgroup$
    – Danu
    Commented Feb 21, 2015 at 12:28
  • 7
    $\begingroup$ Aerodynamics folks like to flip their Y-axis. Let's hope there's never an aerodynamics/astronomer collaboration or we'll all need mirrors to view the data. $\endgroup$
    – tpg2114
    Commented Feb 21, 2015 at 17:37
  • 7
    $\begingroup$ @Danu: Or shows that Chris' assigning of quality scores to his answers was (subconsciously) influenced by their number of votes. ;) $\endgroup$
    – ACuriousMind Mod
    Commented Feb 22, 2015 at 1:41
  • $\begingroup$ So I'm guessing axes like these are fairly common in astronomy? $\endgroup$
    – Emilio Pisanty
    Commented Feb 25, 2015 at 13:59
2
$\begingroup$

So, I took the tour through my old answers and assigned qualities to every one of them as Chris suggested.

Since I couldn't bring myself to assign a 5 to any of my answers, my scale is effectively one point shorter than his original scale, and I also deleted all of the answers I assigned 0 to except for those I couldn't because they were accepted answers, so that data point is also effectively worthless.

I have 2, 116, 84, 132 and 12 answers in the quality categories 0-4, in ascending order, with mean votes of 0, 3.6, 3.7, 5.2 and 4.0, respectively. I might have missed some answers when manually assigning quality scores, but not more than ten, I think.

I find no evident correlation at all:

enter image description here

enter image description here

$\endgroup$

You must log in to answer this question.