7
$\begingroup$

Stack Exchange recently started a collaboration project with the University of Melbourne, in an attempt to improve the automatic detection of duplicate questions.

More information on the project and a call for help from the community can be found here: Project Reduplication of Deduplication Has Begun!

In it, Tim Post mentions that Doris Hoogeveen (me) will post a message on the different meta sites to let you know that we're ready for help. This is it!

As a brief summary, the project is about finding ways to automatically determine if two questions are duplicates, and also to detect the opposite: falsely tagged duplicates. To validate our methods, we need manual judgements from people who really know the data. This is where we need your help. It is impossible to manually annotate all the data from Stack Exchange, and so we have applied several filters to make sure we only show you questions pairs with a high likelihood of being duplicates.

Participation is completely optional, and there's no minimum time requirement. There's a reputation requirement however: only people who have duplicate question tagging rights on the site can participate. That is people with at least 3000 reputation points, or a gold badge. This is to ensure the annotations are compatible with StackExchange's quality standard.

If you are interested in participating in the project, please head over to the annotation interface, which can be found here: http://hum.csse.unimelb.edu.au/se-annotate/

You can log in with your Stack Exchange account, so no sign up is required. The site also contains some more information on the project, which might answer some questions you may have. And I'm also happy to answer them here.

Thank you!

$\endgroup$
22
  • $\begingroup$ Is there any way to get the LaTeX formulae to display? I'm currently using the scripts we normally use for StackExchange chat, but that is a bit buggy for inline formulae. $\endgroup$
    – ACuriousMind Mod
    Commented Nov 11, 2016 at 1:05
  • $\begingroup$ Hi ACuriousMind, I have made an adjustment. The LaTeX should show up correctly now. Could you confirm that? $\endgroup$
    – Monozygotic
    Commented Nov 11, 2016 at 1:20
  • $\begingroup$ Also, I just got offered this question as part of a pair. It's not actually an on-topic question and we have a policy that means we sometimes close other broad book requests as a duplicate of it in order to point people at the resources there while very specific recommendation questions are not closed as a duplicate. Technically, of course, all book requests are a duplicate/subset of this question. How should we annotate in this case? I suspect it might be good to remove this particular question manually from the question pool. $\endgroup$
    – ACuriousMind Mod
    Commented Nov 11, 2016 at 1:23
  • $\begingroup$ I would say annotate them as you would treat them on the site. We will use the data to build a model to automatically tag questions as duplicates, so the main question is: Would you like an automatic system to tag the questions shown to you as duplicates or not? I could remove the question from the pool, but actually the kind of negative feedback on it that you just gave us is very useful. We are not only interested in positive cases, but also in negative ones. $\endgroup$
    – Monozygotic
    Commented Nov 11, 2016 at 1:27
  • $\begingroup$ A bit of confusion: there is a dot in the left of duplicate of and two arrow sign in its right; does that mean the question in the left of the dot is duplicate of the question in right of the arrows? $\endgroup$
    – user36790
    Commented Nov 11, 2016 at 4:20
  • $\begingroup$ Yes! You are the first one to comment on this, but you are absolutely right. $\endgroup$
    – Monozygotic
    Commented Nov 11, 2016 at 4:24
  • $\begingroup$ I've just come across two posts on time-dilations. Well, there is a canonical post on time-dilation in Phys.SE which means every other questions regarding whether it is true or real meaning of time-dilation is actually a dupe of the canonical post. How should such a scenario be handled? $\endgroup$
    – user36790
    Commented Nov 11, 2016 at 4:52
  • $\begingroup$ Do you mean that you know this canonical time-dilation post exists, but it's not one of the two posts shown to you? $\endgroup$
    – Monozygotic
    Commented Nov 11, 2016 at 5:43
  • $\begingroup$ yes; this is what I'm saying. $\endgroup$
    – user36790
    Commented Nov 11, 2016 at 6:02
  • $\begingroup$ Right, I see what you mean now. That's a tricky one. Please pretend you don't know about the existence of the canonical post and base your judgement purely on the two posts shown. $\endgroup$
    – Monozygotic
    Commented Nov 11, 2016 at 6:14
  • 3
    $\begingroup$ What is the criterion for "related, but not duplicates" to be applied? How closely do they have to be related? Obviously, if they share most of the tags they're going to be (very broadly) related, but is that relevant? $\endgroup$
    – Danu
    Commented Nov 11, 2016 at 10:00
  • 3
    $\begingroup$ Sounds like we're training a neural net to replace us :-) $\endgroup$
    – John Rennie
    Commented Nov 12, 2016 at 7:30
  • 2
    $\begingroup$ Hi everyone, lots of questions. How closely do questions have to be related to be tagged as related? If one question contains information that could be interesting to the person asking the other question, even though the questions are not duplicates, then you can tag it as related. Obviously, there's some subjectivity in this judgement, but that's ok. We are collecting multiple annotations per question pair, so if it turns out one pair is tagged differently by multiple people then we know it's a tricky case. $\endgroup$
    – Monozygotic
    Commented Nov 13, 2016 at 0:08
  • 1
    $\begingroup$ The nr of days for this experiment are not pre-fixed. Of course I'd like to get the annotations as quickly as possible, but I also know that people are busy and are helping out voluntarily, so we'll see how it goes. There are some deadlines for me coming up at some point, but they are still quite far off, so there's plenty of time. $\endgroup$
    – Monozygotic
    Commented Nov 13, 2016 at 0:11
  • 2
    $\begingroup$ @AccidentalFourierTransform, that's a difficult questions. Use the criteria you would use on the site. I think the idea is to choose the question with the best wording as the canonical one, or the one that is more general, or the one that has received better answers. That is up to your judgement. $\endgroup$
    – Monozygotic
    Commented Nov 13, 2016 at 0:13

1 Answer 1

Reset to default
2
$\begingroup$

The project is drawing to a close, and I am planning to take down the annotation interface this weekend.

Thank you very much to everyone who helped out, and thank you for all the valuable feedback and suggestions. Hopefully the collected annotations will lead to the development of better duplicate question detection systems, especially for StackExchange.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .