Geoffrey West takes a look at the unknowability of the innards of complex systems, and wonders if big data — and a new big theory to corral big data — could act like a flashlight, revealing the inner workings of financial markets, urban communities, and ecosystems under stress.
Geoffrey West, Big Data Needs a Big Theory to Go with It
Complexity comes into play when there are many parts that can interact in many different ways so that the whole takes on a life of its own: it adapts and evolves in response to changing conditions. It can be prone to sudden and seemingly unpredictable changes—a market crash is the classic example. One or more trends can reinforce other trends in a “positive feedback loop” until things swiftly spiral out of control and cross a tipping point, beyond which behavior changes radically.
What makes a “complex system” so vexing is that its collective characteristics cannot easily be predicted from underlying components: the whole is greater than, and often significantly different from, the sum of its parts. A city is much more than its buildings and people. Our bodies are more than the totality of our cells. This quality, called emergent behavior, is characteristic of economies, financial markets, urban communities, companies, organisms, the Internet, galaxies and the health care system.
The digital revolution is driving much of the increasing complexity and pace of life we are now seeing, but this technology also presents an opportunity. The ubiquity of cell phones and electronic transactions, the increasing use of personal medical probes, and the concept of the electronically wired “smart city” are already providing us with enormous amounts of data. With new computational tools and techniques to digest vast, interrelated databases, researchers and practitioners in science, technology, business and government have begun to bring large-scale simulations and models to bear on questions formerly out of reach of quantitative analysis, such as how cooperation emerges in society, what conditions promote innovation, and how conflicts spread and grow.
The trouble is, we don’t have a unified, conceptual framework for addressing questions of complexity. We don’t know what kind of data we need, nor how much, or what critical questions we should be asking. “Big data” without a “big theory” to go with it loses much of its potency and usefulness, potentially generating new unintended consequences.
When the industrial age focused society’s attention on energy in its many manifestations—steam, chemical, mechanical, and so on—the universal laws of thermodynamics came as a response. We now need to ask if our age can produce universal laws of complexity that integrate energy with information. What are the underlying principles that transcend the extraordinary diversity and historical contingency and interconnectivity of financial markets, populations, ecosystems, war and conflict, pandemics and cancer? An overarching predictive, mathematical framework for complex systems would, in principle, incorporate the dynamics and organization of any complex system in a quantitative, computable framework.
'The emergent is everywhere and nowhere.'
Complexity isn’t a clock. You can’t open one up and see it’s innards. There are no gears and cogs. If there was a way to ‘look inside’, all you’d find would be more complex systems. And those complex systems aren’t connected in purely physical way, made up of computable inputs and outputs: they are united by emergent behaviors: the system manifests it character my acting in ways that are inherently unpredictable, and incalculable. These behaviors arise from the interactions between the components, but reside in none of them. The emergent is everywhere and nowhere.
We have no math for this.
West might as well be saying ‘We need to be able to see into the future . It would be helpful.’ But that doesn’t mean we have a way to do it, or that it is doable at all.
The tempo of modern life has sped up to the point that the future feels closer, and since it’s only a heartbeat away it seems reasonable to imagine being able to glance around that corner and know what is about to transpire. But that’s just a feeling.
'The more we have wired everything into everything else, the less we can know about what will happen tomorrow.'
The future is actually farther away than ever, because we have constructed a world that is the most multi-facted astrolobe, the most incestuous interconnection of global economic interdependencies, the deepest ingraining of contingent political scenarios, and the widest pending cascade of possible ecological side-effects. The more we have wired everything into everything else, the less we can know about what will happen tomorrow.
In essence, West hopes we can create a math that can pile up all the big data and crunch it, in a Borgesian infinity. A machinery as complex as the world it hopes to fathom, allowing us — or at least it— to know everything about everything.
I suspect we will have to settle for something less.
We could start by intentionally decoupling complexity that poses threats. Derivative trading, and credit default swaps are a good example. Efforts by banks and brokerages to diffuse risks, and sharing them with other finance companies leads to increased risk, systemically. When there is a big downturn the risks are amplified, and the cascade leads to huge ‘unintended’ results. The solution to this is not predicting when and how it will happen, but stopping the increased complexity inherent in derivatives and credit default swaps. The only cure for increased complexity is decoupling components of the larger system.
Yancey first introduced me to Geoffrey West and his way of thinking (cities never die, corporations mimic the life/death of humans.)
For a bit more on the topic you can watch Geoffrey West’s TED Talk on the topic.
Richard Florida, Where the Skills Are
Florida looks at US cities and wonders if something big is coming as our cities grow, and as we are concentrating certain skills in different areas. And he cautions that cities can’t do what they are designed to do — efficient creation of ideas and their application — unless we take care of the physical, infrastructure side of urbanism. Since they are meant to be a place for people to interact, we have to make sure that they are social spaces.
But Florida never delves into the post-industrial city, where online interaction is just as critical and deep as off, where cooperation is easier, and chance insights are even more low-cost. More to follow in ‘Liquid City’, my book, in process.
At the risk of putting my fingers in the sausage machine, let me add a touch of nuance:
Winer is ideologically opposed to closed, proprietary approaches like that of Twitter (or, by extension, of Flipboard):
Dave Winer, What I mean by “the open web”
Anyway, here’s what I meant by “open web.”
I meant not in a corporate blogging silo.
If I put stuff in Twitter, the only way to get it out is through a heavily regulated and always-changing API. It will change a lot in the coming months and years. It will certainly narrow more than it expands. I feel very confident in predicting this, because I understand where Twitter is going.
If you put stuff in Facebook, it’s even more silo’d than it is in Twitter.
However, if you put stuff in WordPress, even on wordpress.com, you have full fluidity. You are not silo’d. You can get data in and out using widely-supported APIs that are implemented by Drupal, Movable Type, TypePad, etc etc. At least there’s some compatibility. And in a pinch you could probably move your content to a static website and have it be useful.
If you write in static HTML and RSS, you’re very portable, there will be no lock-in at all.
So to the extent you’re locked in, that’s the extent you are not on the open web. The perfectly open web has zero lock-in. The silos are totally locked-in and therefore not on the open web.
Winer’s complaints are about control of our content: that we should be able to easily manage what we write. It’s a political argument.
But his points fly in the face of innovation, where a Twitter or Quora or Facebook create very different — and not solitary — models of open social discourse, which need to be managed in ways that are different from old school blogging. It’s not every man for himself, anymore. Time is a shared resource on today’s web: our time is not our own, anymore. And that’s largely good.
I liken this problem to the trade offs inherent in living in large cities versus towns or the country. There’s more noise, bigger crowds, and longer lines at the DMV: more things that we can’t control, or where our control is restricted, relative to folks living in bucolic Des Moines.
Only in cities we get superlinear scaling, as Geoffrey West and his colleagues have shown:
Jonah Lehrer, A Physicist Turns the City Into an Equation
When a superlinear equation is graphed, it looks like the start of a roller coaster, climbing into the sky. The steep slope emerges from the positive feedback loop of urban life — a growing city makes everyone in that city more productive, which encourages more people to move to the city, and so on. According to West, these superlinear patterns demonstrate why cities are one of the single most important inventions in human history. They are the idea, he says, that enabled our economic potential and unleashed our ingenuity. “When we started living in cities, we did something that had never happened before in the history of life,” West says. “We broke away from the equations of biology, all of which are sublinear. Every other creature gets slower as it gets bigger. That’s why the elephant plods along. But in cities, the opposite happens. As cities get bigger, everything starts accelerating. There is no equivalent for this in nature. It would be like finding an elephant that’s proportionally faster than a mouse.
I maintain that Twitter, Facebook, and other ‘closed’ systems are really something else: they are dense and complex social systems, more like modern cities than Web 1.0 publishing platforms. And, like cities, there is more going on, less being controlled by specifications like RSS, and the food is better, the music is better, and there is more dangerous sex taking place.
Brian Eno uses the term ‘scenius’ to define the quality of the great cities, their ability to foster deep shared understanding and purpose for large networks of people. This collective intellect arises from messiness at scale, not carefully mediated and clearly defined standards.
Said differently, the best food comes from cities with the highest number of health code violations, and the best art is produced where the largest number of building code infractions are found.
So, if you are looking for clean bathrooms and no traffic jams, stay in Iowa. But it is in cities — dense, loud, unplanned, messy — where the breakthroughs emerge.
Getting back to the specific case, here, let’s look at Flipboard. Flipboard rejects the use of neat-and-tidy RSS, and reaches through the URLs it finds in Twitter to directly paw the text, images, and links placed into articles and posts, and then it chooses what to display based on a proprietary algorithm inside the guts of the app, not based on the publisher’s RSS specification.
Flipboard, Twitter, and other dense, complex social tools create a messier world, one that has superlinear scale. The tradeoff between complete ‘openness’ (or individual control of information and its experience) and superlinear social density is one I am willing to make. And so are all the users of these tools, or should I say, residents of these cities?
- Jonah Lehrer, A Physicist Turns the City Into an Equation