If there's one thing I've consistently argued on this blog, it's that predictions are usually a waste of time and money. Instead, test your assumptions. Don't just “make assumptions explicit.” Look for how you might be wrong, because then you can do something about it.
So how did that play out, the morning after the US Presidential election? Leave aside your horror or elation. This isn't a partisan point. No matter what your politics or feelings about the result, there's a pattern of bad decisions and misjudgment here. And everyone will also forget that pattern within weeks.
With hours to go before Americans vote, Democrat Hillary Clinton has about a 90 percent chance of defeating Republican Donald Trump in the race for the White House, according to the final Reuters/Ipsos States of the Nation project.
The Huffington Post put Clinton's chances at 98%. (98%!)
The HuffPost presidential forecast model gives Democrat Hillary Clinton a 98.2 percent chance of winning the presidency. Republican Donald Trump has essentially no path to an Electoral College victory.
Huffpo also rather sneeringly attacked Nate Silver's 538 for estimating Clinton's chances at a mere 65%.
While I love following the prediction markets for this year’s election, the most popular and widely quoted website out there, fivethirtyeight.com, has something tragically wrong with its presidential prediction model. With the same information, 538 is currently predicting a 65 percent chance of a Clinton victory
As for The NY Times, their final prediction was
“Hillary Clinton has an 85% chance to win”
It's easy to criticize in hindsight. But why do people keep doing this? Why do naive people keep believing this kind of faux-technocratic nonsense? It just leads people to damaging self-delusion, not just in politics but in business and markets.
Elaborate models and data are no defense against wishful thinking. “Big data” does not protect you against many kinds of error. Monte Carlo simulations can be foolish. How could people possibly put a 98% chance on an election that was close to the margin of error in the polls, especially after the lessons of the shock results of Brexit, the Greek referendum and many others?
But they did. Financial markets were bamboozled, for example. Again.
Reuters: Wall Street Elite stunned by Trump triumph.
We need a better way to do this. Instead of models, you need an antimodel, which is what I am developing.
How do we make sense of a week that rocked American public life to the core?
It’s been another appalling, disastrous few days for the GOP establishment in particular. It now appears Ted Cruz is the only contender with even the faintest hope of catching Trump in delegate counts. But Cruz is even more unacceptable to the party leaders than Trump, because is so hostile to current institutions and power structures. Rubio has performed poorly in the latest round of states and is still in danger of losing Florida. As a result, the Republican party is in “meltdown”, “chaos”, “collapsing.” The Bush wing of the party has been utterly routed. How could this happen?
If you look at explanations by insiders, like Romney’s speech, you just can’t help noticing the overwhelming shock and disbelief that someone who doesn’t espouse conservative principles is doing so well out in the country among supposedly conservative voters.
And there lies a major blindspot. As I’ve often argued, it’s assumptions and ways of thinking that get people into trouble fastest, rather than particular facts. They never even notice facts that don’t fit their view of the world.
One of the most important patterns is just this: whether people primarily think in terms of abstract principles. This often tends to go with extra education and jobs which require justification of actions or abstract thinking about rules, such as journalism or law or some kinds of politics. And academia is in many ways all about abstract principles.
So notice that the GOP establishment usually tend to define their position in terms of principles or abstract ideals such as limited government, strict adherence to the constitution, free trade, and usually a tendency to foreign intervention in the service of principles like democracy.
Set aside for a moment what you think of individual policies, whether they are right or wrong, and look at the pattern of thinking here, what people consider correct or appropriate or relevant. Some people find this impossible to do, in isolation from a sense of moral outrage or indignation.
That, however, is a sign it is not just a matter of politics. It’s a matter of ethics as well, which is why so many people come to feel so strongly about it. It’s how you define the good, and the right, and the true, and for many the cause or purpose in life. No wonder people get intense about the issues.
Morality on the established view is a matter of universal principles and objective impartiality.
It’s a pattern that dates back centuries. Kant, for example, summed up morality as “Always act on the maxim that you can, at the same time, will that it be a universal law.” In other words, anything you do should be justifiable in all times and places and contexts. (You notice that this itself is a principle?)
So consider: this necessarily means that your own self-interest or inclinations cannot be in any way a basis for morality. Instead, you are moral insofar as you are impartial or unbiased between groups, and set up things so that they function according to impersonal rules. You tend to see individuals as rational autonomous agents, instead of people from a particular nation or culture, or time, or place, or background.
This view has a lot to be said for it. It is the reason, for example, we don’t see CEOs of large corporations exclusively hiring their cousins or brothers-in-law for top jobs. It is the basis for bureaucracy, in Weber’s sense; large organizations can’t run without following abstract rules and policies and procedures. Modern life depends in large part on restricting clans or extended families or parochial interests. So this way of thinking comes naturally to managers of large corporations or civil service administrators. The finest thing you can be is objective and impartial and unbiased.
Are you nodding your head? It is a view which is often felt to be so common as to be unquestionable. But it is actually comparatively rare. In the world as a whole, probably only 5% or less of people think this way. Even in most developed countries, less than 20-30% of people think of morality or decisions in this way. As Jonathan Haidt put it in The Righteous Mind, such people (like you) are WEIRD, standing for Western, educated, industrialized, rich, and democratic. Researchers..
reviewed dozens of studies showing that WEIRD people are statistical outliers; they are the least typical, least representative people you could study if you want to make generalizations about human nature. Even within the West, Americans are more extreme outliers than Europeans, and within the United States, the educated upper middle class (like my Penn sample) is the most unusual of all.
Instead, most people typically believe morality is a matter of religious revelation, or custom, or tradition handed down through the centuries. Or a matter of honor and reputation. Or most of all, rules that bind together a particular group or community and promote its survival. Group solidarity – loyalty, self-sacrifice, respect for community elders, observing group norms – count for far more than universal principles. The essence of morality is something like loyalty, pulling your weight for the group.
An Inevitable Flashpoint
Where is this difference between ways of thinking most likely to show up and turn into major clashes and mutual incomprehension? Well, take a look at the issues where Trump established himself and rocketed to the top of the field, immigration and torture of terrorists.
If you think in terms of principles, restrictions on immigration, or building a wall, are almost a perfect example of lack of impartiality, of violation of universal principles, of the need to treat everyone in an equal, non-discriminatory, unbiased way. The essence of morality is to be impartial in these cases, and not favor the interests of a particular group like current inhabitants. (You can see Merkel’s policy on refugees in the same light. )
But if you think of morality in terms of care and loyalty and support for your own community or group, or at least a morality that is embedded in a particular set of customs or traditions, it is the perfect example of near-traitorous disloyalty and lack of ethics, of foolish adherence to vacuous abstraction, or a mere cover for the self-interest and malicious intentions of billionaires, or big business.
Again, before you rush to be shocked at one view or the other, stay calm, step back and recognize that they exist, and they help explain not just the lack of communication but the sense of anger and emotional energy right now. Immigration is almost bound to be a flash point between views.
And torture? It’s obviously a violation of universal principle, one of the worst that can be imagined. How could anyone support infliction of pain as a universal idea? But in the other view, it’s more a matter of reciprocity. It may be undesirable, but in a specific context exceptions can be made if it is for the defense of the group and merely responds to unfair actions by the other side, such as acts of terrorism. Treating people in a universal, unbiased way, on principle, simply isn’t important. (Cue gasps from the universal side.)
So the two sides are talking past each other, and simply don’t understand each other. Unfortunately, it’s likely that as a matter of voter arithmetic that universal rectitude may often come a distant second to voter self-interest. But the way of thinking among the elites define this as the very essence of lack of morality.
Incidentally, as the GOP prepares to tear itself apart over the definition of ” conservative”, the establishment view is more or less than the same as classic 19th-century liberalism.
And on the Democratic side? The Democratic party is mostly a coalition of groups who do act in terms of group solidarity – ethnic loyalty, or labor interests, or the interests of sexual orientation groups, plus rich, educated progressives who also define their view in terms of principles, with slightly different emphasis: equality, welfare, government action to pursue rational principles, a different kind of liberalism.
There is a third kind of fundamental view as well, although it more often gets forgotten or ignored, and can reconcile or ameliorate some of these clashes. That is to see morality or decisions as a matter of balances, a golden mean. But that’s a story for another time.
How people think counts.
“From the New Yorker to FiveThirtyEight, outlets across the spectrum failed to grasp the Trump phenomenon.” – Politico
It’s the morning after Super Tuesday, when Trump “overwhelmed his GOP rivals“.
The most comprehensive losers (after Rubio) were media pundits and columnists, with their decades of experience and supposed ability to spot trends developing. And political reporters, with their primary sources and conversations with campaigns in late night bars. And statisticians with models predicting politics. And anyone in business or markets or diplomacy or politics who was naive enough to believe confident predictions from any of the experts.
Politico notes how the journalistic eminences at the New Yorker and the Atlantic got it wrong over the last year.
But so did the quantitative people.
Those two mandarins weren’t alone in dismissing Trump’s chances. Washington Post blogger Chris Cillizza wrote in July that “Donald Trump is not going to be the Republican presidential nominee in 2016.” And numbers guru Nate Silver told readers as recently as November to “stop freaking out” about Trump’s poll numbers.
Of course it’s all too easy to spot mistaken predictions after the fact. But the same pattern has been arising after just about every big event in recent years. People make overconfident predictions, based on expertise, or primary sources, or big data, and often wishful thinking about what they want to see happen. They project an insidery air of secret confidences or confessions from the campaigns. Or disinterested quantitative rigor.
Then they mostly go horribly wrong. Maybe one or two through sheer luck get it right – and then get it even more wrong the next time. Predictions may work temporarily so long as nothing unexpected happens or nothing changes in any substantive way. But that means the forecasts turn out to be worthless just when you need them most.
The point? You remember the old quote (allegedly from Einstein) defining insanity: repeating the same thing over and over and expecting a different result.
Markets and business and political systems are too complex to predict. That means a different strategy is needed. But instead there are immense pressures to keep doing the same things which don’t work in media, and markets, and business. Over and over and over again.
So recognize and understand the pressures. And get around them. Use them to your advantage. Don’t be naive.
“What went wrong?” is the underlying question in David Frum’s much-talked about piece in the Atlantic on the “Republican Revolt”. How could an establishment candidate like Jeb Bush, who was expected to be almost irresistible and has raised more than $100 million now be running at 3-4% in the polls? How can the GOP primary race have been hijacked by a reality TV star at the expense of experienced Governors and Senators?
Let’s leave aside the betting on who will finally get the nomination, or how good or bad Trump is, as most of the media focuses on little else and most journalistic speculation is essentially useless. To be sure, despite his consistent lead, the Donald may not be inevitable as the field thins out and the ‘ground game’ of turnout becomes important.
It’s just that as of now, some of the most powerful, elite and supposedly expert people in US politics look like losers. This is really not where they want to be.
So let’s coolly step back and look at the pattern. How could the establishment miscalculate so badly? What does this tell us about why decisions go wrong? Could other elites have the same problem?
In a nutshell, people refused to see contrary evidence.
Many establishment policies were not popular with the GOP base, Frum says. Less than 17% favored cuts in social security, for example. Most wanted more deportations of illegal immigrants, the exact opposite of a pathway to citizenship.
As a class, big Republican donors could not see any of this, or would not. So neither did the politicians who depend upon them. Against all evidence, both groups interpreted the Tea Party as a mass movement in favor of the agenda of the Wall Street Journal editorial page. One of the more dangerous pleasures of great wealth is that you never have to hear anyone tell you that you are completely wrong.
They could not see things that did not fit in their frame. They could not learn from errors or defeats. The establishment had been shocked at Romney’s loss in 2012, for example.
And yet, within hours of Romney’s defeat, Republican donors, talkers, and officials converged on the maximally self-exculpating explanation.
That meant Republican leaders decided the problem was Romney’s talk about more immigration enforcement alienating Latinos, the very issue where the establishment differed most with their base and where hard evidence of votes to be gained in the center was (Frum says) mostly lacking.
Otherwise, the party yielded on nothing and doubled down on everything. No U-turns. No compromises.
Instead of adjusting to minimize or forestall the chance of a revolt, or finding a smart alternative way forward, the leadership interpreted things in self-serving terms and escalated.
This, of course, is a problem that is extremely widespread and not confined to the GOP. We saw exactly the same thing on all sides in the last midterm elections.
Perhaps the establishment will be able to adapt now that their problem is (you would think) undeniable and it is darkest before dawn for them. Or they can double down again. But serious damage has been done, and some ground rules of US politics – like the importance of raising money – have been rewritten.
Here’s the takeaway. Once again we see in this example that the fundamental problem with decisions is not really bias, or lack of formal rigor, or failure to gather data. It’s that people most often don’t change their mind in response to evidence. Or they fail to adapt until so late in the game that all the choices are bad. That’s what we need to fix, and would save countless billions of dollars and tens of thousands of companies and careers.
The most brilliant investors intuitively realize this. But as this incident demonstrates, most leaders and managers and policymakers do not. They are surrounded by yes-men. They stick with the familiar. They are clever enough to explain away facts which do not fit their narrative.
People get stuck, and persist too long in self-delusion. They fail to adapt and move when they still have the chance. If you can mitigate that, you can do more than most crystal balls could ever do. After all, if the only thing you see in a crystal ball is your own wishful thinking, what good is it?
Peggy Noonan, writing today about the state of US GOP primary race:
But really, what a year. Nobody, not the most sophisticated expert watching politics up close all his life, knew or knows what’s going to happen. Does it go to the convention? Do Mr. Trump’s roarers turn out? Does he change history?
And no one saw it coming.
But the press and tv and political and economic research firms will drown you in speculation and commentary and confident predictions. That’s yet another reason to distrust them, as I keep arguing. Instead, look for leverage, and resilience. Don’t get locked into a convenient narrative. It’s what you can do to change your own thinking and position that counts.
Nearly everything that was expected to happen in the 2016 presidential race hasn’t, and many things that weren’t expected have. The rise of Donald Trump—even that he would run—was not predicted. Nor was the fall of Scott Walker or the weakness of Jeb Bush’s candidacy. Polls have proved to be unreliable indicators of where the Republican and Democratic campaigns are headed. Hillary Clinton’s coronation as Democratic nominee, we were told, was a sure thing. Now she’s sliding toward underdog status.
What a stupid waste. So much air time and column inches on the Republican primary debate, but so little insight. Fox News got record ratings, sure, but because it promised to be a celebrity takedown of Donald Trump. Maybe TV news should try to get Beyoncé or Kim Kardashian to run for the Democratic nomination to help TV ratings at this rate, if it is all just about entertainment value and who looks good on tv.
For all the endless commentary, political journalists are notoriously only obsessed with the horse race aspect, who is up or down as of today, eight months before the main primaries.
The whole debate thing is a bit ludicrous. It doesn’t really test anything much you want in a nominee or President. As if Putin or OPEC are going to care about a witty comeback or a glib sound bite.
What we should do instead is give each candidate a description of the same situation – reports of a missile attack on a Navy ship, for example – with conflicting intelligence, partial information, state dept briefs, staff analysis, stacks of reports only some of which are useful, and give them an hour to make a decision and explain why. Let the candidates ask three questions for additional data or clarifications. Lock them in a room by themselves and make them decide , which is what you want to see presidents do.
The military play war games all the time , and companies have similar tests. It could be done in a fair way.
It’s not the only thing you want in a President, of course. The ability to make a brilliant speech or organize a massive government bureaucracy matter too. But the ability to make decisions under pressure in a crisis is the must-have, the essential factor, the deal-breaker. And if they are uneasy or lack confidence about being asked to make difficult decisions with incomplete information potentially exposed to scrutiny and criticism, they shouldn’t be in the running for a job which is all about precisely that.
An actual test like this wouldn’t happen, of course, because it would be way too risky and revealing. But this current charade of journalists asking “tough” questions and candidates avoiding gaffes is stupid. Politicians are expert at brushing off “tough questions”.
Debates are a network -TV era idea and institution, good for the 1960s and 1970s. There’s no reason why they should always be as central, any more than we should still send out election results by pigeon or steamship like the 1830s. There should be a tougher, better ,smarter way to test candidates in the internet era, mostly by getting them to DO something, not just pass off cliches to journalists. Choices under pressure will show more about ability and character than any podium soundbite.
After all, it’s only the leadership of the free world, the future of the nation, and the life or death of thousands -perhaps millions – of people that depend on this.
I found it hard to get excited about the huge Republican gains in the mid-terms last week. It’s a great story if you focus mostly on the horse-race aspects – who is up, who is down – and these are of absorbing interest to those on the inside of the process, of course. But the actual consequences are limited, so long as the Senate needs 60 votes to get almost anything done (apart from a few arcane parliamentary budget-related matters), and the President still wields his veto.
Instead, what is fascinating is how people have reacted to the Republican victory since. Naturally, there is an extremely strong tendency to spin it so as not to threaten pre-existing views. Take the Democrats first. This is Clive Crook for Bloomberg view yesterday:
Supporters of the Democratic Party have many theories to explain the drubbing they were handed on Election Day. The explanations seem to boil down to one basic proposition, however: Voters are too stupid to know what’s good for them.
Let me say it clearly: The Democratic Party will continue to underperform until it learns to take election beatings a bit more personally.
The sheer variety of theories based on the stupidity of voters is what’s so impressive. For instance, the Obama administration’s record is good, and the economy is finally doing better; but voters are too stupid to see that. Or: The policy record is poor and the economy is screwed, which is the Republicans’ fault for paralyzing Washington; and voters are too stupid to see that.
Paul Krugman, as usual, is top of the list of offenders. He takes the Republican surge as evidence of the success of mulish obstruction, not potential unpopularity or problems with liberal policies themselves.
But the biggest secret of the Republican triumph surely lies in the discovery that obstructionism bordering on sabotage is a winning political strategy.
It wasn’t anything to do with unpopularity or difficulties with Democratic policy, you see.
The same applies to Republican explanations of their own victory. To the Republican establishment, success was largely because of beating back crazy Tea Party primary challengers and maintaining disciplined message control. In other words, it was wholly thanks to … the Republican establishment, not the mood out in the country itself or even primarily a shift right among voters. For many in the Tea Party, the huge victory last week shows just how wrong the Establishment and the media were to predict that shutting down the government last fall would lead to electoral disaster and voter repudiation. The result (apparently) was the opposite, but only the Tea Party has the guts to take on Washington.
So people are perfectly capable of selectively interpreting events to suit themselves. This is no surprise to the average member of the human race, or anyone who has ever encountered an organization.
But here’s the thing: it also shows most people have immense resistance towards learning from evidence. And organizations are even worse. What matters in practical terms is usually not information or events or surprises, but how people assimilate (or more often, refuse to react) to new evidence. People generally don’t learn from evidence in a smooth or rational way.
This does not matter so much if your job is just commentary or opinion, and there is no penalty for getting things wrong. Journalists and politicians can survive wrong prognostications, but not but failing to be vivid or motivating their followers. Anyone who has to make actual decisions, however, cannot afford just to ignore contrary evidence. But the great business and market failures mostly come about because smart, able, senior people pervasively do ignore uncomfortable or opposing facts.