Candy Guessing

Our school had a candy guessing contest for Hallowe’en.  There were three Jars of Unusual Shape, and various sizes.

Jar 1: 108 Candies

Jar 2: 141 Candies

Jar 3: 259 Candies

The spirit of Francis Galton demanded that I look at the data.  Candy guessing, like measuring temperature, is a classic case where averaging multiple readings from different sensors is expected to do very well.  Was the crowd wise?  Yes.

• The unweighted average beat:
  • 67% of guessers on Jar 1
  • 78% of guessers on Jar 2
  • 97% of guessers on Jar 3, and
  • 97% of guessers overall
• The median beat:
  • 89% of guessers on Jar 1
  • 83% of guessers on Jar 2
  • 78% of guessers on Jar 3, and
  • 97% of guessers overall

Only one person beat the unweighted average, and two others were close. There were 36 people who guessed all three jars (and one anonymous blogger who guessed only one jar and was excluded from the analysis). The top guesser had an overall error of 9%, while the unweighted average had an overall error of 11%. Two other guessers came close, with an average error of 12%. The worst guessers had overall error rates greater than 100%, with the worst being 193% too high.

The unweighted average was never the best on a single jar — though on Jar 3 it was only off by 1.  (The guesser on Jar 3 was exactly correct.)

The measure I used was the overall Average Absolute %Error.  The individual rankings change slightly If instead we use Absolute Average %Error, but the main result holds.

US Flu Forecast: Exploring links between national and regional level seasonal characteristics

For the flu forecasting challenge (https://scicast.org/flu) participants are required to predict several flu season characteristics, at national and at regional levels (10 HHS regions). For some of the required quantities — such as peak percentage influenza-like illness (ILI), and total seasonal ILI count — one may argue that national level values have some relationship with the regional level ones. Or, in other words participants may be led to believe that national level statistics can be obtained from regional level ones.

Will Philae land successfully on the surface of a comet?

Background

SciCasters are following the ESA’s International Rosetta Mission and counting down the days of the much-anticipated landing of the Philae on a periodic comet known as Comet 67P/Churyumov-Gerasimenko. The latest news from ESA states that it will deploy the Philae to the comet on November 12. http://bit.ly/1DQEAcy

Market Accuracy and Calibration

Prediction market performance can be assessed using a variety of methods. Recently, SciCast researchers have been taking a closer look at the market accuracy, which is measured in a variety of ways. A commonly used scoring rule is the Brier score that functions much like squared error between the forecasts and the outcomes on questions.

SciCast, Bluefin-21, and GeNIe

Reposted with permission from SciCast forecaster Jay Kominek. You can find his blog, hypercomplex.net here. 

I’m going to assume you’re familiar with SciCast; if you aren’t, that link is the place to start. Or maybe Wikipedia.

There has been a open question on SciCast, “Will Bluefin-21 locate the black box from Malaysian Airlines flight MH-370?”, since mid-April. (If you don’t know what MH370 is, I envy you.) It dropped fairly quickly to predicting that there was a 10% chance of Bluefin-21 locating MH370. Early on, that was reasonable enough. There was evidence pings from the black box had been detected in the region, so the entire Indian Ocean had been narrowed down to a relatively small area.

Unfortunately weeks passed and on May 29th Bluefin-21’s mission was completed, unsuccessfully. Bluefin-21 then stopped looking. At this point, I (and others) expected the forecast to plummet. But folks kept pushing it back up. In fact I count about 5 or 6 distinct individuals who moved the probability up after completion of the mission. There are perfectly good reasons related to the nature of the prediction market for some of those adjustments.

I’m interested in the bad reasons.

Cluster Analysis Data for HPV Questions

One of our active forecasters requested more information about the cluster analysis for the HPV-related questions on SciCast.

Background

The U.S. CDC reports that human papillomavirus (HPV) is the most common sexually transmitted infection in the U.S.  Because some types of HPV are initially asymptomatic but increase the risk of cancer, particularly cervical cancer in women, great effort has been put into vaccinating the population against it.  Two HPV vaccines have been introduced since 2006, and the CDC encourages their use for girls age 11 and older.

Studies of HPV initially focused on the 13- to 17-year-old population, and the CDC estimates that 53.8% of U.S. females aged 13-17 had been vaccinated with at least one dose of an HPV vaccine in 2012, a gain of 0.8% since 2011.  The vaccination coverage varies widely across U.S. States, but some States are similar.  Therefore instead of linking each State to the US average, we put them in clusters.

Cluster Analysis

Clusters of States were created by first analyzing variables that correlate with HPV vaccination coverage in 2011 and 2012. A simple model using those variables for predicting HPV vaccination coverage explained over half the variation among states. To view the state-level variables, including HPV vaccination coverage estimates, open this Google Sheet.

On the most useful variables, states in a given cluster are more similar to each other than to States in other clusters. To create the clusters, I used the mclust and cluster libraries in R statistical software to try several forms of cluster analysis. Results varied somewhat, and we chose to use the five clusters that were relatively easy to interpret and each contained a reasonable number of States.

Cluster Model

The link structure on SciCast is a simple hierarchy:

• US Rate –> {5 Clusters}
• Cluster –> {States In the Cluster}
• Also, US Rate in 2013 –> US Rate in 2014

That means you can forecast cluster rates given US rates (or vice versa), and a State’s expected rate given it’s cluster’s rate.  (Because States are modeled as scaled continuous questions, you cannot forecast cluster given State.)

Based on our model, we have set initial marginal distributions for all 5 clusters, and an initial conditional distribution of Cluster given the the US is in the most likely state. Both are approximately Normal, with the conditional distribution having smaller variance.

Not Known: StateS Conditional On Clusters

Although state HPV vaccination likely is dependent on the cluster to which the State belongs, we do not have a very clear model of the relation between a specific State’s vaccination and its cluster’s vaccination rate. Forecasts of a State rate given its cluster need to be filled in by users with the “Related Forecasts” section after a forecast on a cluster question (Even a forecast that completely agrees with the current market forecast will open up conditional forecasting options.). Hopefully the statistics in the file will help forecasters devise their own ideas!

Links

• Here, find all HPV-related questions on SciCast.
• Google Sheet with Cluster data.

by Ken Olson