The question, A young paraplegic Brazilian, assisted by a neurorobotic exoskeleton, will take the ceremonial first kick at the 2014 World Cup in São Paulo, Brazil. How far will the ball travel? has resolved as “Between 1-3 meters”. Our Brier score was 0.326, better than uniform.
There were 275 forecasts made by 53 unique users. Did you make a forecast? If so, login and check your dashboard.
Also, check out the epic comment stream on this popular SciCast question.
Analysis of the Predictions
Our raw market Brier Score1 (before smoothing or adjustments) was
0.82 0.33, which is better than the 0.88 0.41 that a uniform distribution would have achieved.
We apologize: we previously calculated the score as if the options had no natural order. But of course 0m < 1-3m < 3-5m < …, so we should have used the Ordered Brier.
Here is a histogram of the Brier scores following each individual edit. The strong spike just below 0.1 is presumably @ted’s persistent edits, which turned out to be correct.
The Brier Score (Brier 1950) measures the accuracy of probabilistic predictions. Lower is better. For dates, counts, and other ordered questions, we use the related Ranked Probability Score (Gneiting & Raftery 2007). Both range from 0 to 2 and we report them together as Briers. On a binary question, a 50-50 forecast yields a Brier of 0.5.
There were 120 participants’ comments. Here are a few:
Thanks to everyone who took the time to weigh in on this question. We’ve decided that we’re going to settle the “Between 1-3 meters” option 100%. That is, literally, “the total horizontal length the ball travels from the point of the kicked until it comes to a stop” which is how the question was written. As @DouglasM pointed out, it appears that an extrapolation of the available evidence would produce the same result, which will hopefully make this outcome easier to swallow for anyone losing out on points. Thanks for making this question one of the most heavily forecasted questions on SciCast to date, and for helping us reach a decision during this unexpected turn of events!
I’m having way too much fun with this discussion, but it probably has to come to an end, or at least a judgment, at some point. After thinking some more, it seems that the best choice is to be very literal and call it 1-3 meters based on the distance the ball actually traveled before being stopped and picked up. Since it appears that an “if only it had rolled” extrapolation will produce the same result, the point is moot..
I would like to note that this is the way soccer kickoffs almost always work, as I understand them. The kicker kicks the ball and then the ball stops when it reaches a referee or another player. I have never seen a soccer kickoff where someone kicks it and everyone crowds around until it comes to a complete halt. There’s always a target that you kick it to (either player or ref) and that target stops the ball when the ball arrives. I think the kick here was totally normal and if some forecasters made bets without realizing how soccer kickoffs work, that’s unfortunate, but not worth voiding a question over. Voiding this question would set a bad precedent, as now any time someone doesn’t agree with a question in the future they can complain it was unclear. The role of the fine print is to avoid these misunderstandings and the fine print is clear here – it’s the distance the ball rolls before coming to a stop. The question should resolve 1-3 meters.
@Prieur: “Distance will be the total horizontal length the ball travels from the point of the kicked until it comes to a stop.” Was the ball kicked? Did it stop? Yes and yes. “A young paraplegic Brazilian” – was he young, paraplegic, and Brazilian? “assisted by a neurorobotic exoskeleton” – the exoskeleton is obvious. “will take the ceremonial first kick” – he did. “2014 World Cup in São Paulo, Brazil” – the location and the event are correct. In each of your examples one of these is violated, so it makes sense to void the question in your hypothetical example. The way it came out to be, I don’t see any contradiction to the terms of the question. Unless the kicker wasn’t Brazilian, of course…
Did you like this post? Follow us on Twitter.
1. The Ordered Brier is officially known as RPS.