Joshua Cooper Ramo
Geoffrey West takes a look at the unknowability of the innards of complex systems, and wonders if big data — and a new big theory to corral big data — could act like a flashlight, revealing the inner workings of financial markets, urban communities, and ecosystems under stress.
Geoffrey West, Big Data Needs a Big Theory to Go with It
Complexity comes into play when there are many parts that can interact in many different ways so that the whole takes on a life of its own: it adapts and evolves in response to changing conditions. It can be prone to sudden and seemingly unpredictable changes—a market crash is the classic example. One or more trends can reinforce other trends in a “positive feedback loop” until things swiftly spiral out of control and cross a tipping point, beyond which behavior changes radically.
What makes a “complex system” so vexing is that its collective characteristics cannot easily be predicted from underlying components: the whole is greater than, and often significantly different from, the sum of its parts. A city is much more than its buildings and people. Our bodies are more than the totality of our cells. This quality, called emergent behavior, is characteristic of economies, financial markets, urban communities, companies, organisms, the Internet, galaxies and the health care system.
The digital revolution is driving much of the increasing complexity and pace of life we are now seeing, but this technology also presents an opportunity. The ubiquity of cell phones and electronic transactions, the increasing use of personal medical probes, and the concept of the electronically wired “smart city” are already providing us with enormous amounts of data. With new computational tools and techniques to digest vast, interrelated databases, researchers and practitioners in science, technology, business and government have begun to bring large-scale simulations and models to bear on questions formerly out of reach of quantitative analysis, such as how cooperation emerges in society, what conditions promote innovation, and how conflicts spread and grow.
The trouble is, we don’t have a unified, conceptual framework for addressing questions of complexity. We don’t know what kind of data we need, nor how much, or what critical questions we should be asking. “Big data” without a “big theory” to go with it loses much of its potency and usefulness, potentially generating new unintended consequences.
When the industrial age focused society’s attention on energy in its many manifestations—steam, chemical, mechanical, and so on—the universal laws of thermodynamics came as a response. We now need to ask if our age can produce universal laws of complexity that integrate energy with information. What are the underlying principles that transcend the extraordinary diversity and historical contingency and interconnectivity of financial markets, populations, ecosystems, war and conflict, pandemics and cancer? An overarching predictive, mathematical framework for complex systems would, in principle, incorporate the dynamics and organization of any complex system in a quantitative, computable framework.
'The emergent is everywhere and nowhere.'
Complexity isn’t a clock. You can’t open one up and see it’s innards. There are no gears and cogs. If there was a way to ‘look inside’, all you’d find would be more complex systems. And those complex systems aren’t connected in purely physical way, made up of computable inputs and outputs: they are united by emergent behaviors: the system manifests it character my acting in ways that are inherently unpredictable, and incalculable. These behaviors arise from the interactions between the components, but reside in none of them. The emergent is everywhere and nowhere.
We have no math for this.
West might as well be saying ‘We need to be able to see into the future . It would be helpful.’ But that doesn’t mean we have a way to do it, or that it is doable at all.
The tempo of modern life has sped up to the point that the future feels closer, and since it’s only a heartbeat away it seems reasonable to imagine being able to glance around that corner and know what is about to transpire. But that’s just a feeling.
'The more we have wired everything into everything else, the less we can know about what will happen tomorrow.'
The future is actually farther away than ever, because we have constructed a world that is the most multi-facted astrolobe, the most incestuous interconnection of global economic interdependencies, the deepest ingraining of contingent political scenarios, and the widest pending cascade of possible ecological side-effects. The more we have wired everything into everything else, the less we can know about what will happen tomorrow.
In essence, West hopes we can create a math that can pile up all the big data and crunch it, in a Borgesian infinity. A machinery as complex as the world it hopes to fathom, allowing us — or at least it— to know everything about everything.
I suspect we will have to settle for something less.
We could start by intentionally decoupling complexity that poses threats. Derivative trading, and credit default swaps are a good example. Efforts by banks and brokerages to diffuse risks, and sharing them with other finance companies leads to increased risk, systemically. When there is a big downturn the risks are amplified, and the cascade leads to huge ‘unintended’ results. The solution to this is not predicting when and how it will happen, but stopping the increased complexity inherent in derivatives and credit default swaps. The only cure for increased complexity is decoupling components of the larger system.
Kenneth M Weiss and Anne V Buchanan, What can genes do?
So, it turns out that DNA is, in fact, a great metaphor for business culture, but only after you realize that DNA is not a few hundred off-on switches, but instead a universe of unknowable complexities, that we can interact with, and understand at some abstract cartoonish level, but not control, and never fully comprehend.
Does the flap of a butterfly’s wings in Brazil set off a Tornado in Texas? - Edward Lorenz
A butterfly stirring the air today in Peking can transform storm systems next month in New York. - James Gleick
The Buttefly Effect suggests that the entire ecosystem is a single, swirling, non-linear complex system, and what goes on in one corner of it influences all other parts in unknowable ways.
Scott Rafer riffs on my recent post, The Biggest If Of All, where I suggest that this time it might be different, this time we may have moved into a new era, a new economy: the postnormal. Rafer says it’s just the same old same old:
@stoweboyd This purely academic question gets asked every business cycle. It was being asked on the upside in the late 90s if you recall. In this (not very) regulated financial environment, investment managers figure out how to play new conditions which makes the answer to your question “No” +/- 20%.
Well, logically, just because someone said X would happen 15 years ago and it didn’t doesn’t mean that someone saying X now is wrong. Those are independent events, at least in principle.
My point is something else entirely. We are living in a time where uncertainty is so great that businesses and investors are finding it increasingly impossible to make judgments about where things are headed. Andrew Ross Sorkin recently wrote about this:
The Election Won’t Solve All Puzzles - Andrew Ross Sorkin via NYTimes.com
“Uncertainty” has become the watchword over the last several years for many chief executives, politicians and economists as an explanation — or perhaps an excuse — for the economy’s slow growth, for the lack of hiring by business and for the volatility in the stock market.
“The claim is that businesses and households are uncertain about future taxes, spending levels, regulations, health care reform and interest rates. In turn, this uncertainty leads them to postpone spending on investment and consumption goods and to slow hiring, impeding the recovery,” a group of professors from Stanford University and the University of Chicago wrote in a study that found “current levels of economic policy uncertainty are at extremely elevated levels compared to recent history.” (The professors have created a Web site, policyuncertainty.com, where you can track the “uncertainty” levels.)
If you go look at the other charts — like the European Policy Uncertainty Index — economic uncertainty has been steadily rising since 2007.
We are moving from a world of problems, which demand speed, analysis, and elimination of uncertainty to solve, to a world of dilemmas, which demand patience, sense-making, and an engagement of uncertainty. - Denise CaronSo my point is different. Investors and other business people will find it harder to reason about possible futures because we have moved onto shifting ground. It’s a VUCA world, characterized as increased volatility, uncertainty, complexity, and ambiguity.
As I wrote in July, regarding our blindness regarding the postnormal climate we’ve made for ourselves,
The biggest problem is that people’s thinking patterns are stuck in the old days, and I don’t just mean their expectations about ‘normal’ weather. No, even worse is that people can’t accept the reality that in the post-normal we will never have the luxury of time to assess and then adapt. Linear problem-solving approaches will simply not work anymore.
But this is not a call for more old world leadership, characterized by moving fast, and looking for permanent ‘solutions’ to well-defined and researched ‘problems’. Instead, we need leaders demonstrating the ‘VUCA Prime’ characteristics, as Bob Johansen has styled it.
Denise Caron makes the break between the old world and the new one very clear:
We are moving from a world of problems, which demand speed, analysis, and elimination of uncertainty to solve, to a world of dilemmas, which demand patience, sense-making, and an engagement of uncertainty.
So, in this context, there is no ‘solution’ to infrastructure stress and failure based on more violent weather. We are stuck in a problem space which is fundamentally unsolvable, but we have to try to make sense of this in the context of the larger world.
For example: the financial constraints of our weakened economy mean that we may not be able to repair the interstate highway system, but we might extend and maintain the train system for people moving. Do we have the foresight to disinvest in the highway system? Can we shift from a truck-based logistics system to boats, trains, and airships for long-distance hauling?
We are just as trapped in our thinking as we are in a rapidly changing global weather system, and without leaders with the mindset and skillset geared for the post-normal world, we will never find our way out.
The analysis about weather is paralleled by our inability to logically untangle the financial mess the world is in. And it’s not that we need to get smarter, do more analysis, put more brilliant minds on it: the system is so large, interconnected, and complex that it cannot be understood. It is a complex non-linear system, barreling along as fast as we can fuel it, and it cannot be neatly reduced to a set of smaller, more easily understood parts, unless we actually start disconnecting the parts.
But are we taking steps to disconnect the world’s financial markets? To raise trade barriers, and diminish global supply chains? To require companies to only do business in one country, and to only compete in a single marketplace? To break up vertically integrated multinationals? No. And leaving aside whether this would be a ‘good’ thing in some moral or ideological sense, we aren’t doing it. If anything, the world is growing more interconnected and complex.
At the macroeconomic level, this poses astonishing policy issues, the first of which is seeing the forest for the trees: that we’ve moved into new territory and we have no map. At the microeconomic level, the investor or business leader has a set of tools that used to work, a map that used to show the way, a compass that found north. But they don’t work anymore. They no longer point the way, or suggest that all ways forward are equally uncertain of success.
Specifically with regard to investments in tech, David Lee at SV Angel recently said ‘It has never been easier to start a company, and never harder to build one’, regarding the structural issues in the tech funding world. VC’s don’t see a clear path for a real return on investments in commerce 2.0, games, or apps that rely on Facebook, Twitter, or other platforms. And the result of that uncertainty is being reflected in a decreased amount of later stage investments. This is an echo of the international fund managers I wrote about in the first installment of The Biggest If Of All, many of whom state that uncertainty has never been greater, or of more import in the investment world. So they, like tech VC’s, are holding back, and waiting for a return to normalcy.
But what if it never comes?
We know that the changes we’ve already made to our ecological world will take at least hundreds of years to reverse. Perhaps we’ve turned a similar curve in the economic and policy world. And we don’t know what the world will look like in a hundred years or so, and perhaps there is simply no way to figure out what is going to happen in the next five years, either.
The author of The Black Swan, Nassim Nicholas Taleb, has a new book out — Antifragile: How to Live in a World We Don’t Understand — in which he argues that the world has three sorts of things in it: the fragile, the robust and the antifragile. He was interviewed about the antifragile in New Scientist:
Antifragile: How to make an unstable world strong - Linda Geddes Interviews Nassim Nicholas Taleb
Linda Geddes: In your new book you talk about things being “antifragile”. What do you mean exactly?
Nassim Nicholas Taleb: When you ask people what is the opposite of fragile, they mostly answer something that is resilient or unbreakable - an unbreakable package would be robust. However, the opposite of fragile is something that actually gains from disorder. In the book, I classify things into fragile, robust or antifragile.
Geddes: Can you give me some examples?
Taleb: Nature builds things that are antifragile. In the case of evolution, nature uses disorder to grow stronger. Occasional starvation or going to the gym also makes you stronger, because you subject your body to stressors and gain from them. Another example is the restaurant industry. It benefits from the fact that individual restaurants are fragile by exploiting their mistakes as it tries to figure out why a particular restaurant went bust. Trial and error is an antifragile activity.
Geddes: How is antifragility different from the saying “what doesn’t kill you makes you stronger”?
Taleb: I look at it in terms of systems: situations where what kills me makes others stronger, how the fragility of some parts of the system brings overall benefits. There are good and bad systems organised in terms of whether the system gets stronger or weaker from errors made by an individual part. Every plane crash makes the next one less likely, but every bank going bust makes the next one more likely.
Geddes: How would you make something antifragile?
Taleb: If antifragility is the property of all these natural complex systems that have survived, then depriving them of volatility, randomness and stressors will harm them. Just as spending a month in bed leads to muscle atrophy, complex systems are weakened or even killed when deprived of stressors.
If you want to move away from fragility, you must avoid centralisation and debt and foster aggressive risk-takers who are willing to fail seven times in a lifetime. The economy of the west was initiated through trial and error. We still have a pocket of that in California, where there are small costs of failure and big gains once in a while. The top-down approach blocks antifragility and growth, whereas everything bottom-up thrives under the right amount of stress and disorder.
Geddes: Are you saying that capitalism is good, but that 21st-century capitalism has gone too far?
Taleb: What we do today has nothing to do with capitalism or socialism. It is a crony type of system that transfers money to the coffers of bureaucrats. The largest “fragiliser” of society is a lack of skin in the game. If you are mayor of a small town, you are penalised for your mistakes because you are made accountable when you go to church. But we are witnessing the rise of a new class of inverse heroes - bureaucrats, bankers, and academics with too much power. They game the system while citizens pay the price. I want the entrepreneur to be respected, not the CEO of a company who has all the upsides and none of the downsides.
Geddes: Does all this connect to your black swans?
Taleb: Those are rare events with extreme impacts that lie outside the realm of regular expectations because nothing in the past can convincingly point to their possibility. The global financial collapse is one example (it was a black swan for those who took risks they didn’t understand). Some people have misconstrued my original idea to think that we have to try to predict these events. We can’t, we’ve just got to get out of their way.
I haven’t read the book, so I can’t comment in depth, but I am a deep believer in the premise that interconnecting large and complex systems makes them more — not less — liable to fail. This is why the interconnectednees of the modern world economic system is so dangerous. And I agree that the solution is to disconnect them, intentionally, to decrease risk. Or, using his term, to make systems antifragile.
In reading through the Pew/Elon University Big Data survey analysis, I come away with the sense that I diverge with many others on a basic notion around big data. I don’t believe that big data technology and techniques will end the volatility, uncertainty, complexity, and ambiguity of the post-normal world. Those factors are growing faster than our data exhaust, and our capacity to mine it.
This doesn’t mean that big data collection and analysis is pointless. On the contrary, it may be a critical factor in developing new ways to sense, model, and think about a future growing increasingly opaque.
But some comments, like those of Patrick Tucker, strike me as techno-utopian and unsupportable. He makes that case that Google is already in the business of predicting the small-bore future around issue like flu prevention, based on flu sufferers search queries.
But the passing of the flu bug from person to person is a first-order result of contact. The nature of complex and unpredictable systems is that unforeseeable results arise from second-, third-, and higher-order connections. The public health implications of direct contact have been know for hundreds of years: Google is merely finding a faster way to track the well-understood.
As Tucker argues,
Futurist machines are taking over the job of inventing the future. Their predictions have consequences in the real world because our interaction with the future as individuals, groups, and nations is an expression of both personal and national identity. Regardless of what may or may not happen, the future as an idea continually shapes buying, voting, and social behavior. The future is becoming increasingly knowable. We sit on the verge of a potentially tremendous revolution in science and technology. But even those aspects of the future that are the most potentially beneficial to humankind will have disastrous effects if we fail to plan for them.
The future, alas, is not becoming more knowable.
Unconstrained and dynamic complex systems — like our society, the economic system of Europe, or the Earth’s weather — are fundamentally unknowable: their progression from one state to another cannot be predicted consistently, even if you have a relatively good understanding of both the starting state and the present state, because the behavior of the system as a whole is an emergent property of the interconnections between the parts. And the parts are themselves made up of interconnected parts, and so on.
Yes, weather forecasting and other scientific domains have been benefited by better models and more data, and more data and bigger analysis approaches will increase the level of consistency for weather, but only to a certain extent. There are rounding errors that grow from the imprecision of measures and oversimplifications in our models, so that even something as potentially opaque as the weather — where no one is intentionally hiding data, or degrading it — cannot be predicted completely. In everyday life, this is why the weather forecast for the next few hours is several orders of magnitude better than the forecast for 10 days ahead. Big data — as currently conceived — may allow us to improve weather prediction for the next 10 days dramatically, but the inverse square law of predictability means that predictions about the weather 10 months ahead are unlikely to dramatically improve.
So, consider it this way: Big data is unlikely to increase the certainty about what is going to happen in anything but the nearest of near futures — in weather, politics, and buying behavior — because uncertainty and volatility grow along with the interconnectedness of human activities and institutions across the world. Big data is itself a factor in the increased interconnectedness of the world: as companies, governments, and individuals take advantage of insights gleaned from big data, we are making the world more tightly interconnected, and as a result (perhaps unintuitively) less predictable.
David Cheriton, Arista Networks Founders Aim to Alter How Computers Connect
Yesterday’s Food 2.0 event in NYC was great: a wide variety of people discussing various parts of the emerging open food chain, from the seeds in the ground to dinner in our mouths.
We are headed for the food system equivalent of the bank meltdown.
My interests, as I said in the Open Source Food panel, are mixed. As a web anthropologist, I want to understand the way that innovations spread across this new community, and the role that social tools are playing in spreading ideas and forming a new foundation for a new and open design for food growing, production, and distribution. But I am also motivated — at a partisan level — to help develop an open food chain because of the threats that arise with the closed food chain we now have.
I made a closing argument — and not well — that I think is helpful for people to understand how important an open food system is. Here it is, somewhat cleaned up.
Today’s Food System Is A Lot Like The Financial System, Only In Reverse
The financial system that nearly led to a global financial collapse, and which has caused the largest recession since the Great Depression, is a sprawling, globalized mess so complex that it cannot be managed or understood. One thing we have learned is that the mortgage bond market is a textbook example of how lack of visibility can make a system dangerous to all those involved, even to those who make money from its instability.
Specifically, the housing market worked like this: Home owners acquired mortgages from lending institutions — banks, mostly — and those institutions turned around and sold them off to other institutions, and ultimately (often after many sales and repackagings), the debts were consolidated in to gigantic pools, and then reconfigured into new pools of bonds, theoretically by level of risk.
We know what happened: too much easy credit was offered to people who had high risk of defaulting. This was concealed by the numerous reshufflings of the actual obligations of specific people for specific properties into giant mass blogs of toxic risk. Regulators who were supposed to be monitoring the system as a whole, do not have tools adequate to police the system. In fact, the system is too complex to even determine what should have been monitored, aside from the unhelpful notion that everything should be watched. Those involved in the repackaging had incentives to conceal the actual risks, which they did by design and by actions. And then, when groups of debtors began to default (the mortgage holders, and various hyper-leveraged financial institutions), there was no way to unscramble the mess. Governments have stepped in to cover institutional bets, but we still haven’t see the last repercussions of the Econolypse, and the foreclosures continue.
Why is this like the food system? The world financial system — and the leverage it created — was based on the loss of information at every step in the system, along with implicit trust given to the theoretical authority of regulators and the presumption that large players don’t want to cause harm, necessarily. And the same situation holds in the world food system, with similarly scaled risks. The world’s food is hanging in the balance.
So, in the global food system, the risks inherent in food — not just food safety, but the environmental costs of outsourcing food production to distant lands with unregulated food production — are concealed by distance, and the unwillingness of the players to keep track of and share information. Where did this particular head of broccoli grow? What chemicals were poured on the soil there? Was this side of pork ever allowed to warm above 40º farenheit? The large agribusiness firms bundle together the risks in the food system, and parcel it out in repackaged lots, so when we buy some broccoli at the store — in general — we know nothing about it, really. An ‘enlightened’ chain might mark it as coming from Mexico, or California, but aside from that, we know nothing. And there is no real open marketplace, aside from the choice to defect from the global system and rely on local farmers, which is not a choice open to many. An open marketplace would mean that I could choose one head of broccoli over another based on information about them.
In the final analysis, long food chains with closed information cannot be safe, and create a situation where it is impossible to make informed decisions about the impacts of our food choices. And the companies that have come to control the global food market do not want to gather or provide that information. And it is difficult to imagine that our governments would start to compel them to do so.
The cost of food is rising across the world, as a result of growing populations and various short-term weather or long-term climate problems, depending on your view. This is the stress that is equivalent to the run up to the mortgage/bond disaster: players at every stage in the global food chain will have added incentives to guard their proprietary information and connections, to conceal high risk production practices, and to stockpile foods and information. We are headed for the food system equivalent of the bank meltdown.
The creation of an open food system will have to take place outside of the existing food system, by different groups, and serving the needs of people who have defected. And it needs to happen fast.
There are a vast number of issues related to the development of a food system where critical information is opened up for use by companies and individuals. Open in this sense means that common frameworks for sharing information evolve and are widely used. Food information will of necessity flow in parallel with the movement of food, and the activities surrounding its travel and transformation into food products.
I predict that there will be something like the LAMP stack for this new open food chain: layers in an application framework, or components in a communication framework. Also, this framework will be based on social networks: the relationship of the various agents in the food chain — farmers, distributors, food producers, grocers, and consumers — and their interactions.
The open food system will be social, and is as potentially disruptive to the established closed food system as social media has proven to be for the media world. Low-cost and low-friction software will mean that we can demassify food the way that social tools have demassified media. Supermarket chains might be a lot like newspaper conglomerates, in this model. Yes, we will still need to grow, produce and distribute food, but just as we have increasingly turned to the web to learn about — and influence — world and local events, so too we will turn to an open and social food system, managed online, to learn about and acquire food.
I will be writing more about the open food system, and its software, over the next weeks and months. This is just the start of something big.