I recently listened to a podcast that radically changed my thinking about forecasts. In this post I explain the relevance of predictions in technology and PR, and about armchair superforecasters who can do a radically better job than experts. The cool thing is, apparently the skill can be learned by just about anyone (are you listening, Gartner?)
Predictions in PR, Technology and Society
Tracking the direction of tech is important for our work in PR. I recently ran an educational session for our team about this. Some of you may have seen my post about surfing tech trends.
Looking beyond just PR, technology prognostication is big business. The high priests of IT (e.g., Gartner, Forrester and IDC) get nice fees for their reports and forecasts.
Yet predictions are often wrong – even those made by the experts. See this story in The Register which pokes fun at Gartners’ Hype Cycles. And sometimes we see other articles that point out prediction fails, like this NY Times piece on pundit accountability.
So who cares if predictions in all those articles don’t pan out? Why is it so important to make accurate predictions in tech, and for PR to try to guess correctly too?
Just think about it. If we all had better crystal balls:
- PR firms could choose sectors that have growth potential; we could craft better pitches and plans
- Lawmakers and regulators could use the insight to prevent bad stuff and leverage the good for society
- Investors and VCs could make better bets
- Enterprises could invest in the right IT roadmap
- Vendors could make smarter R&D choices
- Gartner and other analysts could avoid snarky articles
The list goes on and on.
But how many times have we heard “this will be the year of_____ (fill in the blank: IoT, AI everywhere, VR)”? How many times does that ball get punted? And what about the blowback from poor tech design, policies, data and security vulnerabilities?
So I was intrigued when I heard the podcast, on the Ezra Klein Show (he is a NY Times writer, and often has superb guests and topics on the podcast).
The Rise of the Superforecasters
The episode description said: “Can we predict the future more accurately? It’s a question we humans have grappled with since the dawn of civilization… It’s also the question that Philip Tetlock, a psychologist at the University of Pennsylvania and a co-author of ‘Superforecasting: The Art and Science of Prediction,’ has dedicated his career to answering.“
Guest host Julia Galef interviewed Tetlock as Ezra was on paternity leave. She has her own podcast, Rationally Speaking, and wrote the book “The Scout Mindset: Why Some People see Things Clearly and Others Don’t.”
The podcast explained that Tetlock pitted his group of amateur prognosticators against academics and career intelligence analysts in a forecasting tournament in 2011. Tetlock’s team won by such a large margin that the government agency behind the competition decided to study the superforecasters and their methods, to see what they could learn.
He shared with Julia and the podcast audience insight into the mindset and approach of superforecasting. It doesn’t necessarily take genius or years of experience (remember, it is the so-called experts who can have spotty records at predictions).
Philip claimed that the skill can be taught to college students in a matter of hours. How is this even possible? I’ll try to summarize here, but suggest you check out the podcast and read Tetlock’s book to learn more.
In a few words, it gets to being a “flexible fox” vs. an “inflexible hedgehog.” The latter tend to be constrained by ideology. It also is about trying to be less bombastic, and more accurate – the former works better if you want to be a firebrand or a talking head on TV, vs. someone who calls shots in a less dramatic way.
As Tetlock says, it is important to be “integratively complex and qualify your arguments, howevers and buts and all those, a sign that you recognize the legitimacy of competing perspectives.”
The less successful forecasters dig themselves into a position that they want to defend, perhaps because it is their trademark issue. The more successful ones are more open and set conditions that might support a counter POV. Superforecasters use “comparison classes” and adopt “outside views;” they examine benchmarks to see how other similar situations played out.
Most of the podcast focused on big picture global economic and geopolitical topics vs. the world of technology. But Julia and Philip finished off with an example, about how a superforecaster might approach a question about technology advancement. Of course, that grabbed me.
The resulting back and forth is fascinating and shines a light on how a superforecaster might approach the question. I copied an excerpt from the episode transcript:
JULIA GALEF: Are we going to get to full self-driving cars by the year 2025? The kind of car where you can really just take a nap and let the car drive you to work through busy city streets… So do you have any thoughts on how a superforecaster might start thinking about that question?
PHIL TETLOCK: The first thing they would do is they would pick holes in it.
JULIA GALEF: In the question?
PHIL TETLOCK: Yeah, they’d pick holes in your question.
JULIA GALEF: OK.
PHIL TETLOCK: It would be like you’re talking to a contract lawyer. And they’d say, oh, Julia, my God, this is so underspecified. OK, you mean anywhere in the world?
JULIA GALEF: Yes, yes, I do. Yes.
PHIL TETLOCK: OK, so you have more of a shot of that happening in Singapore because they are pretty far advanced with the infrastructure, and they have a different legal system… technology is never going to be perfect. Given the way the U.S. tort law is set up and given the dynamics of U.S. politics, it would be very difficult to imagine it happening by 2025 in the U.S.
Could it happen in certain authoritarian or semi-authoritarian systems that are technocratic, maybe in Dubai or in Singapore or places like that? If that’s the question, now I’m saying, hmm, OK, maybe 25%. But now take a nap — hmm, not so sure. I think even there, the laws would be you have to be conscious, and you probably shouldn’t be drunk. There would still be laws and a sentient observer observing, at least in principle, in charge. But yeah, that’s how they think. They think in very detailed terms. And they think about boundary conditions. And it’s not just the technology, it’s the politics and the law, right?
JULIA GALEF: I guess this is how a lot of forecasters go awry, is they’re thinking of one specific kind of scenario. I was thinking about the U.S., and I was only focused on the technology itself and not on the legal framework surrounding that deployment of the technology, et cetera. And so, I can, a little bit, empathize with the experts in your tournament who, when the question is resolved and they were wrong, they kind of protest, oh, but that’s not what I was thinking of when I said 75% chance yes. The thing I was thinking of was blah, blah, blah. But that is part of the game, is you have to be thinking about all the factors. And I guess that’s what the superforecasters are good at.
The Ezra Klein Show
