[reposted from Darwin, July 2003 issue, courtesy of the Wayback Machine]
At last: The socialization of the enterprise leads to intelligence rising to the top.
IN 1999, I WROTE a piece about Abuzz, a then-newly funded start-up that had developed a fascinating expertise management solution. The product was called BeeHive, and was based on insights about emergent behavior in groups that grew out of research into the interactions and behavior of social insects, hence the name. The premise of BeeHive was to enable an online interaction between members of a community so that when someone encountered a difficult problem the system would provide either an answer to the question or a pointer to someone likely to be able to help. Later on, Tacit Knowledge Systems and others (such as Lotus) would also field systems based on these same ideas. Abuzz was acquired by the New York Times while I was researching them, and its technology more or less disappeared from view, although it supposedly powers some online NYT communities.
For example, within a large professional services company a consultant might to looking for assistance with a customer’s request, such as a European telephony regulatory issue. Now this is more or less the heartland of knowledge management, and there have been countless KM solutions that are based on the premise that knowledge can be strip mined from people’s heads, and stored in digital form in a system for later use. But these expertise management solutions take a different tack. They are based on an adaptation, not prediction.
Karma and Whuffie: Self-organizing Swarmocracies
In these systems, answers to hard questions are handled on demand, in (more or less) real time. Expertise management systems build a profile of users’ interests, and ask other users to rate the quality of answers or other assistance each user provides, so that over time a measure of expertise can be associated with each user relative to various categories of questions.
The guy who consistently answers questions about Java to others’ satisfaction quickly becomes — within the system, and the social network that it comes to embody — a Java expert. And in the most advanced of these solutions, with a rise in community-based merit, the experts’ ratings of others are weighted more strongly than non-experts. The basic principle is that these online aggregations of experts are self-organizing and self-monitoring: the system acts as a matchmaker, and mediates by introducing those who have questions with those who have the answers. And the most active and knowledgeable are acknowledged as such, both by getting recognition based on ratings and gaining more weight in ratings of others merit.
The emergence of social order from emergent properties of merit-based social interaction is a potent self-organizing principle, and is likely to form the foundation of all adaptive social tools in the future. This principle has been named many times: at Slashdot it is called “karma.” Corey Doctorow called it “whuffie” (in his science fiction work, Down and Out in the Magic Kingdom). I favor the term “swarmth” because it gets at the heart of what is happening — swarm intelligence leading to filtering out the dumb, and intelligence rising to the top.
Other Recent Sightings
Swarm intelligence is showing up in other interesting technologies. CompanyWay has taken the principle of swarmocracy and applied it to the problem of organizational decision making.
Consider, as a working example, the issues surrounding a company’s product planning processes. How does management know that it is evaluating the market data correctly? How to understand client requests for improvements? How much money should be allocated to the various product lines? Who gets to decide which issue, and how do the various issues impact each other?
In the typical company, these product decisions are traditionally handled in a top-down fashion over a determined period of time: senior management allocates funds to various functional groups, product line managers aggregate information from various sources and hand over product requirements to engineering or design groups, and so on. But in a rapidly changing, perhaps even incomprehensible market, how can management be sure that these piecemeal decisions come back together for a winning or even acceptable product? Or said in other ways, how can you be sure that you are right?
CompanyWay’s technology approaches this problem from the bottom up, likewise over some period of time, but less likely a bounded period. A large group, perhaps hundreds of interested parties would be brought into a CompanyWay swarmocracy, and let loose like a bag of ladybugs. For example, all the members of the product process: designers, engineers, packaging experts, marketing, salespeople and even (dare we say it?) customers. Every member of the swarm may start with exactly the same swarmth — the same rating, the same weighting on votes, etc. &3151; but relatively quickly this changes, as individuals begin to post ideas into a shared collaborative space, and others begin to rate the value of the ideas, and then still others rate the value of people’s comments about ideas.
The administrator of the swarm can exercise a kind of jujitsu-like control on the Brownian motion going on, by assigning CompanyWay-managed merit points to specific ideas that have been raised, and assigning a merit award date to the idea. This attracts members of the swarm to spend more time (and thought) on the ideas valued more highly by the administrator. When the merit award date comes, let’s say a few weeks later, the merit is distributed on some defined number of those whose contribution to that idea was rated most highly by the swarm. Members can also acquire “wisdom” (so-called) by selecting a course of action for a particular issue — continue, abandon, fund, for example — that accords with what the administrator ultimately decides to do. As members acquire more swarmth — either through ratings, merit or wisdom — the weight of their votes increases. At a point identified by the administrator, swarm members can accumulate enough swarmth to become a sub-administrator of a sub-swarm devoted to a particular issue or group of issues.
It’s Getting Swarm In Here
Swarm logic takes some getting used to, even as a theoretical case, let alone actually revamping business processes. Jettisoning the top-down, command-and-control structures that most businesses are built on, and accepting a socialized decision making process — even one remotely controlled by the administrator’s distribution of merit points or assigning higher swarmth to specific individuals at the start — takes courage, and a belief in the swarm.
Eric Bonabeau, a researcher in swarm intelligence and founder/CTO of Icosystems, when asked about the apparent reluctance of business management to accept the notion of bottom-up, emergent solutions to thorny problems, said, “Managers would rather live with a problem they can’t solve than with a solution they don’t fully understand or control.”
Top-down decision-making in corporations relies on a belief in accurate prediction: a small number of minds, perhaps only one, assimilating serially a large body of data and producing an single, grand analysis that leads to a large number of people performing their smaller, circumscribed roles. Think of a symphony, written and conducted by a single person, and an orchestra playing exactly how it is told.
Bottom-up decision-making is based on a belief in adaptation: a large number of minds, perhaps thousands of minds, working in parallel, assimilating small bits of information and producing many small analyses, that lead to others being influenced in their analyses, and so on. A fractal decision-making process, where the activities at one scale directly reflect the activities at another.
This is not an orchestra — at least not a traditional one, anyway. (I read about a bottom-up orchestra, whose name escapes me, that operates without a conductor. Its rehearsals are much longer than traditional ones, but its music is nonetheless top-flight and its members are unwilling to leave the swarm to make “factory music,” as one member put it.) This is more like a jazz ensemble, building the music one riff at a time through improvisation.
It’s The Socialized Enterprise
I believe that the basic social wiring in the human mind makes swarming easy, easier than the traditional alternative. I have my own horror stories about stupid decision-making processes, poor product planning and autocratic business governance leading to disaster. Perhaps it is exactly those experiences that lead me to conclude that not only is swarming easier, but that it is better.
It is better to socialize decisions with those that are involved in them — not as a mealy-mouthed “empowerment” exercise, but as a means to get the best ideas out in the light of day, to allow all to make their contributions and raise their concerns, and to recognize what is good based on merit and not on personage. Louis Brandeis once said, “Light is the best disinfectant,” and socializing critical business decisions in an open forum based on merit, and not on politics, is a powerful model of business.
This cannot be effectively managed in large groups without the support of swarmth-based technologies. The complexities of balancing the variables of thousands of peoples’ comments about thousands of other peoples’ ideas and so on are too big for the human mind to grasp. Cognitive psychologists have established the limits of the human understanding of social groups, and we are configured for groups of no more than 150, the size of a proto-historical clan, where people can keep track of who likes who, who is whose cousin, and what does she like to eat. Beyond that, we lose track.
Obviously, swarms larger than 150 can be effective, but we need to rely on technology to do the heavy lifting: keeping track of merit, rating and even (perhaps) deciding who is wisest in groups so large that no one really knows everyone involved.
As a technocrat, I can envision a single swarmth infrastructure underlying a suite of swarm intelligence applications, so that people’s swarmth can persist over projects and be applied in different settings. Or that swarmth might become convertible, or fungible: movable between different systems. Or that swarmth could become a personal asset, linked to your digital identity, like your e-mail address or AIM screen name.
The socialized enterprise is coming. It is emerging in feedback-based blog networks that companies are starting to use internally to manage planning and knowledge. It is emerging in swarm intelligence technologies like CompanyWay. It is being rolled out in expertise management solutions like Tacit. However packaged or implemented, swarm intelligence is likely to force a reevaluation of how companies are organized and managed — and it’s long overdue.
Note: Since writing this column, I learned that CompanyWay has been acquired by AskMe, a leading enterprise knowledge management technology vendor. Most of the features of CompanyWay are subsumed by AskMe capabilities, and AskMe plans to support the principles that drove the development of CompanyWay. Existing clients will be supported indefinitely, and transtioned into future releases of AskMe.