Seven Guidelines for Better Forecasting

Nice summary by longtime colleague and arch argument mapper Tim van Gelder. “The pivotal element here obviously is Track, i.e. measure predictive accuracy using a proper scoring rule.” If “ACERA” sounds familiar, it’s because they were part of our team when we were DAGGRE: they ran several experiments on and in parallel to the site.

Tim van Gelder

“I come not to praise forecasters but to bury them.”  With these unsubtle words, Barry Ritholz opens an entertaining piece in the Washington Post, expressing a widely held view about forecasting in difficult domains such as geopolitics or financial markets.  The view is that nobody is any good at it, or if anyone is, they can’t be reliably identified.  This hard-line skepticism has seemed warranted by the persistent failure of active fund managers to statistically outperform dart-throwing monkeys, or the research by Philip Tetlock showing that geopolitical experts do scarcely better than random, and worse than the simplest statistical methods.

More recent research on a range of fronts – notably, by the Good Judgement Project, but also by less well-known groups such as Scicast and ACERA/CEBRA here at Melbourne University – has suggested that a better view is what might be termed “tempered optimism” about expert judgement forecasting. This new attitude acknowledges that forecasting challenges will always fall on…

View original post 443 more words

Advertisements

Comment notifications temporarily disabled

A new SciCast ad campaign has created ~1,000 registrations per day for the past couple of days.  That has doubled our forecaster community and created a lot of activity, which is great.  But it also generated a lot of email notifications for users who had opted to receive updates for new comments, and more email is not always great.

After a dozen or so complaints and a review of some comments, we have disabled email notifications until we add some more controls.

Continue reading

Decision Analysis Journal Article: Probabilistic Coherence Weighting for Optimizing Expert Forecasts

We’re excited to announce that the Decision Analysis Journal has published Probabilistic Coherence Weighting for Optimizing Expert Forecasts about some work last year related to DAGGRE.

It’s natural to want to help forecasters stay coherent as we ask related questions. For example, what is your confidence that:

1.      “Jefferson was the third president of the United States.”

2.      “Adams was the third president of the United States.”

People are known to be more coherent when these are immediate neighbors than when on separate pages with many unrelated questions in between.  So it’s natural to think it’s better to present related questions close together.

We found that’s not necessarily a good idea. On a large set of general knowledge questions like these, we got more benefit by allowing people to be incoherent, and then giving more weight to coherent people.  At least on general knowledge questions, coherence signals knowledge. We have yet to extend this to forecasting questions.

We found other things, too – cool and interesting things.  Here’s the abstract, but be warned, it gets technical:

Methods for eliciting and aggregating expert judgment are necessary when decision-relevant data are scarce. Such methods have been used for aggregating the judgments of a large, heterogeneous group of forecasters, as well as the multiple judgments produced from an individual forecaster. This paper addresses how multiple related individual forecasts can be used to improve aggregation of probabilities for a binary event across a set of forecasters. We extend previous efforts that use probabilistic incoherence of an individual forecaster’s subjective probability judgments to weight and aggregate the judgments of multiple forecasters for the goal of increasing the accuracy of forecasts. With data from two studies, we describe an approach for eliciting extra probability judgments to (i) adjust the judgments of each individual forecaster, and (ii) assign weights to the judgments to aggregate over the entire set of forecasters. We show improvement of up to 30% over the established benchmark of a simple equal-weighted averaging of forecasts. We also describe how this method can be used to remedy the “fifty–fifty blip” that occurs when forecasters use the probability value of 0.5 to represent epistemic uncertainty.

Read the article!

Christopher W. Karvetski, Kenneth C. Olson, David R. Mandel, and Charles R. Twardy. Probabilistic coherence weighting for optimizing expert forecasts. Decision Analysis 2013 10:4, 305-326

At Decision Analysis | Local PDF  ]

Sneak Peek: Sample Questions

The team is working very hard and we’re getting close to launch! We’re excited to share a sneak peek at some of the questions you will see…

Here’s an example:

What will the average arctic sea ice extent be for September 2014?

Some background: Passive microwave satellite data reveal that, since 1979, winter Arctic ice extent has decreased about 3 to 4 percent per decade. Arctic sea ice typically covers about 14 to 16 million square kilometers in late winter, and reaches a minimum of about 7 million square kilometers in mid-September.  However, recent minima have all been below 6 million square kilometers, with 2012 at only 3.4 million square kilometers.

And, another question:

What percent of managed honey bee colonies in the US will be lost during the 2013-2014 winter?

Some background: Estimates of winter loss for managed honey bee (Apis mellifera) colonies are an important measure of honey bee health and productivity. Last year’s loss of bee colonies was 31.1% which is slightly higher than the previous 6-year average loss of 30.5 percent.

There will also be an opportunity to make forecasts in questions that are “linked” where the probability of something happening in one is affected by the other. For example, we’ll also ask:

How many billions of pounds of almond meat will be harvested in California in 2013?

This is linked to the previous question because almonds depend on bees for pollination; therefore bee colony health will affect agricultural yield, and may affect the amount of almonds harvested.

If you haven’t already, be sure to sign up at www.SciCast.org to be notified the week of December 2 when we launch. We’re also looking for people to help create and edit forecast questions! Send a note to support@scicast.org if you have questions you’d like to see on the site.

SciCast presented at the Northrop Grumman Information Systems’ University Symposium

Members of George Mason University’s C4I Center presented SciCast and two other C4I projects at the Northrop Grumman Information Systems’ University Symposium in McLean, VA this week.  Dr. Charles Twardy (SciCast Principal Investigator) and Dr. Tod Levitt (SciCast Project Manager) discussed the SciCast market, and recruited potential question-writers and forecasters.  There was steady interest and good discussion both of the public SciCast market and other potential uses of the technology.  Our special thanks to Ludwig Tokatlian, Lolo Penedo, and the Future Technical Leaders program for the invitation.

Welcome to the SciCast blog!

SciCast is a government funded research project to forecast the outcomes of key issues in science and technology. SciCast is based on the idea that the collective wisdom of an informed and diverse group is often more accurate at forecasting the outcome of events than that of one individual.

Technically, SciCast is a prediction market. Prediction markets can be used to forecast the outcome of a wide variety of topics and are used today in large corporations and governments to understand the likelihood of meeting key performance metrics, quantify risks that may jeopardize operations, and better understand industry trends.

Unlike other prediction markets you may have heard of like Intrade or Inkling Markets, SciCast will create relationships between forecast questions that may have an influence on each other. For example, we may ask a question about the volume of sea ice in the Arctic in a given month. We may also ask a question about average temperature in this same locale or other influencing metrics. SciCast will learn from its participants how strong of a relationship these questions have to each other and will adjust their outcomes accordingly. This means if the SciCast participants forecast the average temperature will be higher in the Arctic, we’ll adjust the likelihood that the level of ice will decrease in that same time period.

SciCast will be a community-driven initiative. Participants will nominate questions and we will facilitate a process to get those questions published. Once a question is available, SciCast participants will wager their “SciCa$h” to make a forecast: the amount they wager depending on how much they want to influence the collective forecast. And unlike a survey, participants can change their mind at any time and increase or decrease their wager on a particular outcome. In this way, SciCast is a real-time indicator of what our participants think is going to happen.

After the answer to a question is known and made public, participants who answered correctly will be awarded SciCa$h based on their wager. The more correct forecasts a participant makes, the more SciCa$h they earn and the more influence they’ll have in other forecasts.

Our goal is to get thousands of participants from around the world. We’re currently reaching out to Professional Societies, Universities, and interest groups to solicit their participation. If you are part of an organization you think would like to participate, please let us know and we will introduce ourselves.

SciCast will be launched in early December, but you can pre-register now at http://scicast.org. If you’d like the SciCast participants to make forecasts on questions you’re interested in, you can also submit candidate questions now at http://signup.scicast.org/question_form.html