2025 RationalWiki 'Oregon Plan' Fundraiser

There is no RationalWiki without you. We are a small non-profit with no staff—we are hundreds of volunteers who document pseudoscience and crankery around the world every day. We will never allow ads because we must remain independent. We cannot rely on big donors with corresponding big agendas. We are not the largest website around, but we believe we play an important role in defending truth and objectivity.

Fighting pseudoscience isn't free.
We are 100% user-supported! Help and donate $5, $10, $20 or whatever you can today with PayPal Logo.png!
Donations so far: $9223.37Goal: $10000

You're good at statistics

Jump to navigation Jump to search

You're good at statistics

Is there any way to "measure" the accuracy of a prediction of a ranking of items? I ask because I'm interested in this claim by Andy. Presumably, it's not just about position ("Rick Santorum is the only item, of ten, that retained position, therefore the prediction was 10% accurate"), but also about the order - the relative positions of the items in the prediction and the result.

ONE / TALK09:09, 16 August 2011

That is indeed not a trivial problem: if you haven't it defined a metric beforehand you can almost always chose one which makes your prediction looking especially good.

But here is my take: if we look at the nine common names in both list, we can say:

  1. Andy didn't get the winner right. Generally, that is the most important part of the prediction - think of English elections, where the candidate with the relatively most votes wins. Assuming an equal probability for each one to win, it was an 11% probability for an successful prediction, much higher of course if you ignore Huntsman and other footsies...
  2. He got the top two right, but in the wrong order. In an English election, that would be a bad prediction but if you claim that the first two will go on with their candidacy, this could be seen as a success - a monkey had a chance of less than 3% to do so.
  3. But is you say: it's important to get the names of the top four candidates, as these will proceed, a monkey had a chance of 17% to be at least as good as Andy, who got three names right...

So, Andy's predictions are neither abysmal, nor splendid.

larronsicut fur in nocte15:47, 16 August 2011