Mi3: Today, we’re exploring a study titled “Compromising Improves Forecasting” with Crowd Cognition Lab Principal Investigator and Group Director Bahador Bahrami. This research builds on a (literally) million dollar question: can we predict the future? Previous research, popularised by Philip E. Tetlock and Dan Gardner’s book Superforecasters, has shown that collective intelligence can make more accurate forecasts than experts. To start, could you tell us what you have done here? What is this compromise about?
Bahador: Compromise brings a whole new approach: when a group of friends try and discuss who will win the gold at the next Olympics, they discuss and may finally reach a consensus – where everyone agrees. Alternatively, they may still disagree but if they have to place a bet, they will somewhat reach a compromise – some bet distribution that represents everyone’s voice. This is what we did for collective forecasting: we looked not at individual superforecasters, but teams of them!
Mi3: Interesting. So does this really differ from other collective forecasting then?
Bahador: The principle is very different: it isn’t just about putting individual predictions together mechanically; it’s about harnessing the collective intelligence of a group. When people work together and reach a compromise, they’re pooling diverse perspectives and information, which often leads to more accurate predictions than if their opinions were simply averaged together irrespective of who has been talking to whom. This is incredibly relevant in today’s world where decisions need to be made quickly and accurately, often in complex and uncertain environments.
Our results clearly show that when forecasters compromise, the accuracy of their aggregated predictions improves significantly. And this isn’t just a one-time effect; the advantage of compromise holds up across different time frames as the event approaches.
Mi3: Can you explain how you tested this?
Bahador: Absolutely. We analysed five years’ worth of data from the Good Judgment Project, which was led by the team of Philip Tetlock in Harvard. This is a great resource from a series of competitions that involved thousands of forecasts on geopolitical events. What makes our approach stand out is that we didn’t just look at individual predictions. Instead, we focused on how groups of forecasters were organised together. We used this social structure of “who was working with who” to guide our methods to define what compromise is. Rather than averaging the data from everyone with everyone else, we restricted the averaging to the people who worked in the same group and then combined those compromise opinions.
Mi3: The data sound impressive indeed. How is it working with open data?
Bahador: What’s particularly exciting is that the data we used is publicly available, which means other researchers can access it, verify our findings, or even build on them. This openness is vital for advancing our understanding of forecasting and improving the methods used in various fields, from economics to public policy.
Mi3: Can you give us a concrete example of how this might work in a real-world setting?
Bahador: Picture this: you have 50 analysts trying to predict whether a country will enter a trade agreement. Each one makes a prediction based on their own insights, and you end up with a wide range of probabilities. If you average all those predictions, you get what we call the “wisdom of the crowds,” which usually outperforms just relying on a few experts.
But what we have shown and the Good Judgment Project and several other groups as well, is that if you let these 50 analysts discuss their predictions in small groups of five, then combine their predictions after these discussions, the accuracy improves even more.This challenges a common belief in the collective intelligence field—that independent opinions are always better because they avoid groupthink.
But we’re proposing something different: instead of just averaging all 50 predictions, focus on the group compromises. After discussing their reasoning, if each group reaches a consensus and you then combine these 10 group predictions, the forecast becomes much more accurate. The group discussions help cancel out individual biases and errors, leading to sharper, more reliable predictions.
Our current study also found that this approach consistently improves accuracy as the event gets closer, which is crucial for timely decision-making. After all, predicting a trade agreement months in advance is far more valuable than getting it right just days before it happens.
Mi3: That’s a great example. Why do you think these findings are particularly relevant for industries and policymakers?
Bahador: The implications for industries and policymakers are massive, I think. In fields like finance and public health, accurate forecasting is crucial for planning and decision-making and our findings could be a game-changer. Encouraging teams to reach a compromise can lead to sharper forecasts, which ultimately result in better decisions and outcomes. Imagine in financial markets, where predicting trends can have huge economic impacts—using a compromise-based approach could reduce risks and lead to smarter investment strategies.
And it’s not just theory: we’ve already proven that this approach works on a small scale, and it’s a cost-effective way to boost forecasting accuracy. The open data from the Good Judgment Project is just one real life example. By partnering with industry leaders and policymakers, we could put these models to the test and help refine their forecasting methods. The potential for impact is massive.
Mi3: Finally, on a more personal note, what about this research changed your perspective?
Bahador: This research really shifted my perspective on the power of collective decision-making. I have researched interactive collective intelligence for about 15 years now, but often having to do with established facts. It is quite something to see that even when it comes to predicting the future, many heads are indeed better than one!
Check out the original study and Bahador Bahrami’s profile.
References:
Ferreiro Dardo, N., Deroy, O., & Bahador, B. (2023). Compromising improves forecasting. Royal Society of Open Science, 10221216, doi.org/10.1098/rsos.221216