Drift into failure, p.31
Drift Into Failure, page 31
If a regulator cannot regulate a complex system, then what can it do? Will a regulator always be caught behind the curves of self-organization and emergence, holding a bag of obsolete rules that came from less evolved systems? Languages of compliance and regulation can perhaps be juxtaposed against those of coevolution and counter-evolution. Rather than a regulator, complex systems should have a co-evolver/counter-evolver. This must be an organization that has the requisite variety not only to have an idea of the complexity of the operational organization (and thus has to co-evolve with how that organization evolves). It should also have requisite variety to counter-evolve. At least in its theories or models, it should be able to generate alternative outcomes to the small steps that get made by the operational organization.
Safely inspection has an important role to play in complex systems. But inspections can be conducted only on parts or sub-systems. A whole complex system cannot be inspected, only parts or sub-systems can be inspected. After all, if a whole complex system can be inspected, it would mean that the inspector (or group of inspectors) can not only achieve a complete description of the actual system (in which case it wouldn't be complex) but that they also have some ideal description against which they can compare it. These are Newtonian commitments (achieving a complete description of the world and comparing that against a description of an ideal world) that do not work for complexity. Safety inspection, however, should not be about just the parts or their failure rates or probabilities. In itself such findings say very little about the trajectory a complex system may be on, and they can easily be rebutted by operational organizations who can show successful and safe continued operations despite parts being run to failure (see Chapter 1).
This is also why the idea of safety oversight is problematic. Oversight implies a big picture. A big picture, in a sense of a complete description, is impossible to achieve over a complex system. Not only is that computationally intractable, complex systems (as said many times before in this book) evolve and change and adapt the whole time. Nailing its description down at one moment in time means very little for how it will look a next moment. If the big picture of oversight, however, implies a sensitivity to the features of complexity and drift, then it might work. Oversight can try to explore complex system properties such as interconnectedness, interdependence, diversity, and rates of learning and selection that go on inside the complex system. This is perhaps the type of oversight that people inside the operational organization are not capable of because of the locality principle (actors and decision-makers in the organization only see their local interactions). But for a regulator it means learning an entirely new repertoire of languages and countermeasures: from componential, determinist, compliance to co /counter-evolving complexity.
Inspection of parts, then, should relentlessly pursue the possible interconnections with surrounding parts and systems, even those that are external to the sub-system or even organization (like caribou herds outside the Prudhoe bay pipeline). These interconnections can be functional (where parts have a functional or known connection to other parts) but in complex systems the interdependencies are more often non-functional. Caribou have nothing functionally to do with oil transportation, and insulating foam on a fuel tank on lift-off has nothing to do functionally with re-entry of the Space vehicle. The interdependencies are non-functional, and it is unlikely that they can all be made visible if an inspector only looks at existing descriptions of the system (drawings, presentations, documentation, visual and physical inspections). Exploring these interdependencies is a matter of listening to multiple stories, even from those whose responsibilities seem to lie outside of the parts in question. Remember from Chapter 1 that in times of crisis, all correlations go to 1. This means that in times of crisis, these people's responsibilities may well suddenly be affected by a part failure way outside their functional area.
Drifting Into Success
Rather than setting goals and plotting out the road to get there (or keeping a system on a narrow legally-regulated road), we may want to think about ways to enhance the creativity and the diversity of the operations we run or organize or manage or regulate. Complexity says that we need to see strategy (and, ultimately, safety and accidents) as emergent phenomena, not as the logical endpoints of linear organizational journeys that can be easily retraced afterward or foreseen beforehand. Rather than working on the safety strategy, or the accident prevention program, we should be creating the pre-conditions that can give rise to innovation, to new emergent orders and new ideas, without necessarily our own careful tending and crafting. In complex systems there are a thousand ways (or more, many more) in which small steps can become big events. No strategy tinkered together by a smart designer (or team of designers) in the organization itself or the regulator can foresee and prevent them all. It is possible, with an approach that promotes creativity and diversity, that a system might even drift into success.
From a Paperclip to a House
A 26-year-old Montreal man appears to have succeeded in his quest to barter a single, red paper-clip all the way up to a house. It took almost a year and 14 trades, but Kyle MacDonald has been offered a two-storey farmhouse in Kipling, Saskatoon, Canada, for a paid role in a movie.
MacDonald began his quest last summer when he decided he wanted to live in a house. He didn't have a job, so instead of posting a resume, he looked at a red paper-clip on his desk and decided to trade it on an internet website.
He got a response almost immediately – from a pair of young women in Vancouver who offered to trade him a pen that looks like a fish. MacDonald then bartered the fish pen for a handmade doorknob from a potter in Seattle. In Massachusetts, MacDonald traded the doorknob for a camp stove. He traded the stove to a U.S. marine sergeant in California for a 100-watt generator. In Queens, N.Y., he exchanged the generator for the "instant party kit" – an empty keg and an illuminated Budweiser beer sign.
MacDonald then traded the keg and sign for a Bombardier snowmobile, courtesy of a Montreal radio host. He bartered all the way up to an afternoon with rock star Alice Cooper, a snow globe and finally a paid role in a Corbin Bernsen movie called Donna on Demand. "Now, I'm sure the first question on your mind is, 'Why would Corbin Bernsen trade a role in a film for a snow globe?' MacDonald said. Well, Corbin happens to be arguably one of the biggest snow globe collectors on the planet."
Now, the town of Kipling, Sask., located about two hours east of Regina with a population of 1,100, has offered MacDonald a farmhouse in exchange for the role in the movie. MacDonald and his girlfriend will fly to the town next Wednesday. "We are going to show them the house, give them the keys to the house and give them the key to the town and just have some fun," said Pat Jackson, mayor of Kipling. The town is going to hold a competition for the movie role.
MacDonald said: "There's people all over the world that are saying that they have paper-clips clipped to the top of their computer, or on their desk or on their shirt, and it proves that anything is possible and I think to a certain degree it's true."10
Drifting into success is possible because in complex systems the source of emergent phenomena like safety and accidents is not the individual component. These phenomena do not appear as a logical endpoint of a linear organizational journey. They result, rather, from the complex interactions among multiple, diverse, interconnected and interdependent agents who mutually affect each other. This means a couple of things:
❍ We need to attend to relationships that can help bring fresh perspectives to the fore, that can help novelty emerge. While we can influence who gets to talk to whom by certain organizational and institutional arrangements, and influence the weighting of their voices when they do get to talk, we should never believe that we can perfectly circumscribe this. In complex systems, there are many more relationships and changes in relationships than we can predict or keep track of. Unforeseen people will talk and influence decisions in ways that lie beyond detection or control.
❍ Building the preconditions for diversity, however, is one strong candidate for managing complexity. Greater diversity of agents in a complex system leads to richer emergent patterns. It typically produces greater adaptive capacity. Seeking a diversity of people, cultures, (technical) languages and expertise (and even age and experience) will likely enhance creativity More stories get told about the things we should be afraid of, more hypotheses may be generated about how to improve the system.
❍ Small changes can lead to large events. Changing the language of organizational procedures connected to risk in one way or another (calling something "in-family" or "run-to-failure") can seem to be no big deal. But it can produce reverberations which over time constrain or condition what people elsewhere in the system will see as rational courses of action.
❍ Emergence is certain, be we cannot be certain what will emerge. Designing solutions to complex system safety problems is not likely to be successful by itself. Of course, component re-design is always one option (such as the IV ports in the example that started off Chapter 3: they can be redesigned to make certain complex system events less likely). But solutions to safety problems evolve; they don't just follow the path of the designer. The world in which they are supposed to work, after all, is too complex and unpredictable.
Complexity, Drift, and Accountability
Remember the question from Chapter 1: Who messed up here?' Newtonian logic allows us to answer it relatively easy. But a Newtonian narrative of failure achieves its end only by erasing its true subject: human agency and the way it is configured in a hugely complex network of relationships and interdependencies. The Newtonian identification of a broken part becomes plausible only by obscuring all those interdependencies, only by isolating, mechanizing, or dehumanizing human agency, by making it into a component that zigged but should have zagged. This is both existentially and morally comforting. It is existentially comforting because it allows us to pinpoint a cause forbad things. If we take away the cause or somehow isolate it, we can rest assured that such bad things won't happen again. It is morally comforting because it allows us to place responsibility in the hands of the people whose risk management was deficient: the Newtonian story allows us to hold them accountable.
Since ideas about systemic accident models were first published and popularized, however, system safety has been characterized as an emergent property. Safety is something that cannot be predicted on the basis of the components that make up the system. Accidents, too, have also been characterized as emergent properties of complex systems. They cannot be predicted or explained on the basis of the constituent parts. Rather, they are one emergent result of the constituent components doing their (normal) work. Drifting into failure is possible in an organization where people themselves suffer no noteworthy incidents, in which everything looks normal, and everybody is abiding by their local rules, common solutions, or logics of action.
Recall that emergence means that the behavior of the whole cannot be explained by, and is not mirrored in, the behavior of constituent components. No part needs to be broken for the system to break. Instead, the behavior of the whole is the result – the emergent, cumulative result – of all the local components following their local rules, and of them interacting with each other in innumerous ways, cross-adapting to each others' behavior as they do so. Going from a model of broken components, and the moral and existential satisfaction it may give us, to an understanding of complexity and its much fuzzier, less determined idea of accountability, can be frustrating and difficult. It would almost seem as if complexity gives no room for morality, that it is in itself amoral. Scott Snook concluded as much when he had studied the shoot-down of two U.S. Black Hawk helicopters in 1993. The two helicopters, carrying UN peace keepers, were downed erroneously by two U.S. fighter jets in the no-fly zone over northern Iraq:
This journey played with my emotions. When I first examined the data, I went in puzzled, angry, and disappointed – puzzled how two highly trained Air Force pilots could make such a deadly mistake; angry at how an entire crew of AWACS controllers could sit by and watch a tragedy develop without taking action; and disappointed at how dysfunctional Task Force OPC must have been to have not better integrated helicopters into its air operations. Each time I went in hot and suspicious. Each time I came out sympathetic and unnerved. If no one did anything wrong if there were no unexplainable surprises at any level of analysis; if nothing was abnormal from a behavioral and organizational perspective; then what ...?
Snook's impulse to hunt down the broken components (deadly pilot error, controllers sitting by, a dysfunctional Task Force) was tempered by its lack of results. In the end he came out "unnerved," because there was no way he could clearly identify a "cause" that preceded the effect. The most plausible stories of the incident lay outside dominant Newtonian logic.
With an outcome in hand, its (presumed) foreseeability suddenly becomes quite obvious, and it may appear as if a decision in fact determined an outcome; that it inevitably and clearly led up to it. In a complex system, the future is uncertain. Knowledge of initial conditions is not enough because the system can develop in all kinds of unforeseeable ways from there on. Also, complete knowledge of all the laws governing the system is unattainable. This is true for the nonlinear dynamics of traditional physical systems (for example, the weather) but perhaps even more so for social systems. Social complex systems, composed of individual agents and their may cross-relationships, after all, are capable of internal adaptation as a result of their experiences with each other and with the system's dynamic environment. This can make the possible landscape of outcomes even richer and more complexly patterned. As a result, a complex system only allows us to speculate about probabilities, not certainties.
This changes the ethical implications of decisions, as their eventual outcomes cannot be foreseen. Decision-makers in complex systems are capable only of assessing the probabilities of particular outcomes, something that remains ever shrouded in the vagaries of risk assessment before, and always muddled by outcome and hindsight biases after some visible system output.
In complexity and system thinking, not only is there no clear line from cause to effect, there is also no obvious symmetry between them as in a Newtonian system. In a complex system, as we have seen, an infinitesimal change in starting conditions can lead to huge differences later on (indeed – having an accident or not having one). This sensitive dependency on initial conditions removes both linearity and proportionality from the relationship between system input and system output. The asymmetry between "cause" and "effect" has implications for the ethical load distribution in the aftermath of complex system failure. Consequences cannot form the basis for an assessment of the gravity of the cause. Trivial, everyday organizational decisions, embedded in masses of similar decisions and subject to special consideration only with the wisdom of hindsight, cannot be meaningfully singled out for purposes of exacting accountability (for example, through criminalization) because their relationship to the eventual outcome is complex, non-linear, and was probably impossible to foresee.
If we adjudicate an operator's understanding of an unfolding situation against our own truth, which includes knowledge of hindsight, we may learn little of value about why people saw what they did, and why taking or not taking action made sense to them. What is unethical or a mistake to one, may have seemed perfectly rational to somebody else at the time a(particularly to the one actually doing the work). This should give some pause for thought about what is ethical to do in the aftermath of a drift into failure. Imposing one normative view onto everybody else Could easily be seen as unethical, as unjust, as unreasonable. If we take a story that pretty much challenges almost everybody's moral sensibilities, we can see how complexity and accountability might play out. This is the story of the drift into failure of Enron, which ended up as the biggest bankruptcy ever in the United States in 2002.
Enron’s Drift Into Failure
Enron had its humble beginnings in natural gas. The production and distribution of natural gas had always been seen as a poor cousin of the petroleum industry. In fact, natural gas had long been wasted or discarded as an unwanted by-product of the production of oil. As its usage for heating and energy increased during the 1960s, its production and distribution remained essentially in government hands. Government set production targets and distribution Contracts, as well as consumer prices. The industry was totally regulated, and attracted nobody who was interested in dynamic business management or interested in making money. At the time it was quipped that you could get by in the natural gas industry with making one or two decisions a year. With the onset of deregulation in the 1980s, however, the industry started attracting players who were more inclined to make quick deals and pursue large sums of money. Ken Lay, who had worked his way up through the energy industry and ended up in Houston at the head of the natural gas production and distribution company HNG-InterNorth, saw to it that the name was changed to the more manageable Enron, and hired Jeff Skilling, a McKinsey consultant who had helped HNG in the past.
Skilling largely disregarded – indeed had an active distaste for – the messy details involved in executing a plan. And he did not see that as his role, it wasn't the job of the executive. What thrilled him was the intellectual purity of an idea, not the translation of that idea into something implementable. The challenge facing the natural gas industry was pure and simple indeed: the relationship between sellers and buyers was entirely distorted (because of regulation, Ken Lay would add). Gas up to then had been traded under long-term contracts between producers, pipelines, and local utilities, under prices set by the government. The 1980s saw most of the trading diverted to spot markets, where gas changed hands frantically at the end of each month. The uncertainty inherent in this was the basic problem for everybody. Gas had been seen as dull and static (a network of pipes through which it traveled across the country at leisurely speeds was really the major capital investment to be made), but, in a changing environment it actually represented quite unruly technology A sudden cold spell in the' Northeast Could cause prices to rise overnight, which would hurt consumers. A wave of warm weather, in contrast, could push prices down again, which would hurt producers and distributors. Even with a surplus of natural gas, big industrial users could never be guaranteed to have enough of a supply from one month to the next. Offering such guarantees was not interesting to gas producers, because at fixed prices, they might not be able to deliver it, or could lose money if they did. As a result, gas was not seen as a reliable fuel.
