Emergent Behavior and the End of Management - 2

The first article in this series discussed a Wall Street Journal article which proclaimed the end of management as we know it.  The craft of management was developed to organize thousands of workers to build pyramids, produce cars, refrigerators, and many other products without which modern life would be impossible.

Breaking everything down into simple, repetitive tasks made it easier to hire and replace workers who didn't need particularly complex skills.  This type of organization excels in doing the same thing the same way ten zillion times, but it squashes innovation and collapses utterly when presented with something entirely new.

Another Way

There's another management style which is called "emergent behavior."  In an emergent system, each individual operates independently of everyone else but acts according to a well-designed set of internal rules which take account of others' behavior.  Consider this description of a termite colony:

The cathedral termite, found in parts of Australia, is capable of creating mounds for the colony well over 10 feet high. Individual cathedral termites are just standard-looking bugs - head, thorax, abdomen, legs, and so on, with a tiny little primitive brain. But when combined with others of its species, the cathedral termite is capable of constructing a huge, complex hive to house the colony. Unlike human building projects, however, there is no foreman, no plan, and it's unlikely that any termite even knows what it is helping to create. [emphasis added]

How is this possible?

The answer lies in the fact that sometimes, a system can provide more complexity than the sum of its parts - leading to what scientists call "emergent behavior."

The behavior of the termite colony as a whole emerges from the individual actions of millions of individual termites.  As with GM, no termite possesses enough brain power to understand the termite mound as a whole, but somehow all the jobs get done.  Where's the org chart?  Where are the manuals of policies and procedures?  How do hordes of simple bugs implement such complex behavior?

Hundreds of thousands of spiders collaborated to build a giant web in a Texas state park.  Cooperating to build a shared 200-yard web seems to increase their catch of insects, but if this is true, why aren't cooperative spider colonies more common?

Scientists have researched such issues for many years and a few answers are emerging.  We now know enough about how ants learn enough about what other ants are doing to organize their food gathering efforts extremely efficiently.  You can even download a program which simulates the food-collection part of an ant colony's behavior and experiment with various parameter settings.  We don't yet know enough about the inner workings of an ant's brain to be certain that the ants are using exactly the same food finding rules as the simulation, but the simulated results match up pretty well with what ants are observed to do in the field.

An emergent system is far better at responding to unexpected events than a managed system because it's largely self-organizing.  Ants need a lot of food, so food gathering is the default activity - if an ant isn't doing anything else, it's finding food.  Drop a rock on an ant hill, however, and the ants stop gathering food and scurry around until all their tunnels are fixed.

Do they have a plan for handling natural disasters?  How does each ant know what to do?  Are some ants identified as "first responders?"  It's pretty clear that the ant colony's management structure is both flexible and efficient.

Start-ups and Emergent Behavior

All business start-ups run on a variant of emergent behavior.  Unlike a queen ant who focuses only on eating and laying eggs, the entrepreneur has a pretty good idea of the overall corporate vision.  He doesn't have time to write all the procedure manuals a big firm would have and he doesn't have enough money to hire layers of managers to keep everything running smoothly.

That's a good thing, however.  Unlike a big firm whose sales cycle has largely settled down to a repeatable process, the entrepreneur doesn't really know what his customers want until he tries to sell them something.  Early customers give feedback which the start-up uses to improve the product, and the improvement cycle continues until they succeed or run out of money and fold.  This is the pattern which has played out in start ups like Google, Facebook, Twitter, Microsoft, Ford Motor Company, RCA, Motorola, and many other ventures large and small.

The important point is that the entrepreneur doesn't have time to tell each person in the hierarchy what to do and he doesn't have the classical management pyramid which tells everyone what to do.  The entrepreneur has to convey the vision to everyone and rely on each individual ant to work with the other ants to get the job done.  That's why start-ups try to have their technical people meet with customers as much as possible - the entrepreneur's vision is all very well, but what customers think is a lot more important.

There's a limit to this, however.  Assuming success, the start-up grows to the point that the owners have to start using some of the tools in the management toolbox.  Accounting, purchasing, payroll, and other functions simply must operate in an orderly manner or someone will go to jail for fraud.  At some point, the techies have to be walled off from customers because they'll spend too much time on the concerns of individual customers and product development suffers.  That's when the dreaded "Customer Service Department" emerges.

This transition generally takes place somewhere between 100 and 500 people, and not all entrepreneurs survive.  Many of them have to be kicked out in favor of conventional management once the business settles down enough for a normal management style to be effective.

Open Source Software

The open source software movement has given rise to some of the purest examples of long-term emergent behavior extending far beyond the usual limit of 100 people.  In the early 1970's, Richard Stallman of MIT suggested that all software should be free and that software developers ought to work for the sheer love of it.

I have to confess to having been among the many who thought him crazy, but he turned out to be entirely correct.  Many software projects such the Gnu Emacs programmable text editor, the MySQL database, and the Linux operating system are maintained and extended by swarms of intelligent developers who work with each other closely enough to figure out what to do next while exercising their individual skills.

These developers share the overall project vision and collaborate with each other to contribute the bits and pieces which make it work.  The community is held together by a shared vision and by status gained by furthering the vision.  Individuals who have a track record of having good ideas or of developing really good code gain stature and are listened to more attentively than others, just like ants who are able to find food influence others.  An open source project is the purest example of a market-driven meritocracy visible today.

There are tens of thousands of open source projects, most of which are hardly used at all beyond a few devotees, but popular projects have tens or hundreds of thousands of users.  The open source Apache web server is the most popular server in the world.  Its devotees constantly improve it to keep ahead of commercial competition and the price can hardly be beat.

Where Management Meets Emergence

Some open source projects such as MySQL and Linux give rise to commercial firms.  Red Hat Linux made a successful business out of packaging the free open source Linux operating system and selling support services.  Similarly, MySQL spawned a company which sells database services and support but keeps faith with the open source community by giving the software away for free to anyone who's willing to do without support.

The MySQL company was purchased by Sun Microsystems and, when Oracle bought Sun Microsystems, Oracle ended up owning the company that sells support for a free but highly capable competitor to Oracle's flagship database product.

MySQL and Red Hat have worked very hard to stay in the good graces of the open source community.  As the queen ant has no way to compel any individual ant to serve the hive, there's no way to compel anyone to work on an open source project.

To keep all those intelligent ants foraging away, Red Hat and MySQL have had to play fair.  In particular, they have to follow the open source convention and make any improvements and bug fixes they develop available for free.

It's hard for conventional managers to see how companies can make money doing that, but they have a huge advantage over most software companies - hordes of motivated programmers add new features and fix bugs for free.  This more than makes up for the fact that perhaps 90% of their users pay nothing.  Even these "freeloaders" find bugs and make suggestions, and some join the unpaid labor force.

These companies represent a fascinating hybrid of emergent behavior and managed behavior.  Red Hat and MySQL, then Sun, and now Oracle, have to use classical management to pay the rent, sweep the floors, meet payroll, and manage product development.  On the other hand, they also have to keep myriads of independent-minded developers happy and collaborating effectively.

Big Business and Emergent Behavior

IBM, which is in many ways the archetype of a button-down managed organization, has made effective use of open source software.  IBM worked with Red Hat to develop a version of Linux with which IBM replaced a proprietary operating system they had developed.

Why would freewheeling independent developers want to work with blue-suited IBM?  IBM brought something highly desirable to the party: as a result of years of high-end research unavailable to basement freelancers, IBM's file management system was much more robust than what Linux had at the time.  IBM donated this software to the cause, thereby reassuring the open source community that IBM planned to play fair by contributing it's share of enhancements.

Imagine how hard it must have been for IBM's first open-source visionary to talk management into giving away software that had cost them vast sums to develop!  On the other hand, having swarms of unpaid developers competing to find bugs in their operating system for free represented considerable savings for IBM once their business model swung away from renting software to consulting about software.

IBM also funded the Jakarta foundation which offers a suite of free software development tools.  This eventually forced Microsoft to give away its Visual Studio development environment instead of charging money for it as in the past.  Microsoft also gives away the Express edition of SQL Server to try to stem loss of market share to free databases such as MySQL.

Since it switched its emphasis from mainframes to consulting, IBM has come to agree with Mr. Stallman's view that software should be free - but you have to hire expert IBM consultants at top dollar to operate it for you, of course.

The concept of free software threatens to drive a stake through the heart of Microsoft, a fact that has not been lost on Microsoft's management.  Microsoft's early heavy-handed attempts to keep businesses and governments from using open source software aroused considerable ire in the community, but even Microsoft seems to have decided to get along with selected emergent organizations.

It's becoming clear that for smallish software development projects, a company can hire programmers off the internet and put together short-term development teams much as musicians cooperate to put together short-term musical groups for gigs.  Managing such ad-hoc projects is a bit of a challenge, but a number of virtual organizations support software-based businesses where none of the developers have met any of the others.  This sort of dynamic collaboration bothers conventional managers who prefer to be able to "reach out and touch someone" slaving away in the cubicle farm.

There are problems with the emergent style, of course.  Unlike ants, humans have egos and personal agendas which can get in the way of progress.  The New York Times reports:

A $500 million initiative for independent research promised by BP, which was to be awarded by an international panel of scientists, has become mired in a political fight over control. [emphasis added]

Responding to the oil spill is the sort of catastrophe for which the emergent style ought to be well suited.  BP provided research funding, but the ants who want to hand out all this lovely moolah are locked in a battle over who's to be queen.  The box at the top of the org chart generally gets to make a decision and can fire those who won't participate; emergent organizations find it hard to hammer out decisions particularly when there's big money or big egos involved.

The Coming Hybrid

Some successful businesses have been founded on the concept of selling support for software which was developed by swarms of unpaid developers.  IBM has made effective use of open source software and its development teams to cut costs and gain competitive advantages.

As more and bigger businesses learn to collaborate with emergent organizations, they'll eventually work out the best ways to combine the strengths of classical management with respect to carrying out large, complex projects such as the Apollo moon landing with the flexibility and rapid response of the emergent systems which launched the first versions of eBay, Twitter, and Facebook in a few intense weeks or months of work.

The era of conventional management isn't over - we still need to keep the lights on, and emergent behavior hasn't yet demonstrated that it's good at that - but the future belongs to organizations which combine the advantages of both.

The third, and final, article in this series ought to explain precisely how to do that.  Unfortunately, if we knew how to do that, we, not the Google Guys, would be squiring ourselves around the world on our own private 767.

You'll have to figure it out for yourself but it's worth the effort - the potential rewards are immense.