Drift into failure, p.28
Drift Into Failure, page 28
The edge of chaos is what is called a critical state, or, in an explicit reference to thermodynamics, a dissipative state. While poised in a critical or dissipative state, a system is not in equilibrium and can be easily perturbed. In this state, the many components that make up the complex system never quite lock into place, yet hardly ever dissolve into an entire chaos. For example, by purchasing a faster Coast Guard aircraft to go after jet-bound smugglers across the Caribbean, there will be a response. The smugglers' jet may fly at night, or change routings more often. Perturb the system again, and there might be a completely different response, and one at a different scale – such as the opening of routes through West Africa. This is one very characteristic property at the edge of chaos: Small responses are common, big responses are rare. But big responses are possible, and what they are can be almost impossible to foresee.
Arriving at the edge of chaos is a logical endpoint for drift. At the edge of chaos, systems have tuned themselves to the point of maximum capability. Mixing the base chemicals in a cup and smearing them across scratches and cracks and gouges in the foam covering of the Space Shuttle's external fuel tank was one such response to production pressures. Those actions got the tanks returned to service quickly by making evidence of any maintenance and possible flight safety problems go away. Production was served, and production had been called for (incentivized in various ways, even). The system had tweaked itself to maximum capacity in a constant, dancing interaction with the political, medial and technical environment in which it operated. It was in a critical state, which suddenly allowed a big response to the small, little actions that maintenance workers had applied hundreds of times. A large event happened on 16 January 2003: the separation of a chunk of iced-up foam so large that it managed to penetrate the heat shield on the wing's leading edge, which would later lead to the loss of Columbia while on its re-entry. Drift into failure, in these terms, is about optimizing the system until it is perched on that edge of chaos. There, in that critical state, big, devastating responses to small perturbations become possible. Large events are within the space of possibilities. Drift doesn't necessarily lead to failure. At least not until it does.
References
1 Laszlo, E. (1996). The systems view of the world: A holistic vision for our time. Cresskill, NJ: Hampton Press.
2 Perrow, C. (1984). Normal accidents: Thing with high-risk technologies. New York: Basic Books.
3 Perrow, C. (1984). Ibid.
4 Perrow, C. (1984). Ibid.
5 Seaways. (2010, July). London: The Nautical Institute, pp. 25—7.
6 See, for example, Billings, C.E. (1996). Aviation automation: The search for a human-centered approach. Hillsdale, NJ: Lawrence Erlbaum Associates.
7 Leveson, N.G, (2006). System safety engineering. Back to the future. Cambridge, MA: Aeronautics and Astronautics, Massachusetts Institute of Technology.
8 Mackall, DA. (1988). Development and Fight Test Experiences with a Flight-Critical Digital Control System (NASA Technical Paper 2857). Lancaster, CA: National Aeronautics and Space Administration Dryden Flight Research Facility.
9 Leveson, N.G. (2006). Ibid.
10 Leveson, N.G. and Turner, C.S. (1993). An Investigation of the Therac-25 Accidents. IEEE Computer, 26(7), 18—41.
11 Rose, B.W (1994). Fatal dose: Radiation deaths linked to AECL computer errors. Montreal, QC: Canadian Coalition for Nuclear Responsibility.
12 Leveson, N.G. (2006). Ibid.
13 United Nations Security Council (1). Security Council, 4317th and 4318th meeting, condemns illegal exploitation of Democratic Republic of Congo's natural resources (UN Press Release SC/7057). New York: United Nations.
14 Peat, F.D. (2002). From certainty to uncertainty. The story of science and ideas in the twentieth century. Washington, DG: Joseph Henry Press.
15 Capra, F. (1982). The turning point New York: Simon & Schuster.
16 Capra, F. (1982). Ibid, p. 69.
17 Stapp, FLP (1971). S-matrix interpretation of quantum theory. Physical Review D, 3.
18 Griffiths, P. (2009, September 8). Climate talks must agree amotion CO2 cap – UK adviser. London: Thompson Reuters Foundation Alertnet.
19 Intergovernmental Panel on Climate Change (2007). Fourth Assessment Report (AR4) of the IPCC. More than 450 lead authors, 800 contributing authors, and an additional 2,500 reviewing experts from more than 130 countries contributed to AR4.
20 Wang, G. and Eltahir, E.A.B. (2002). Impact of CO2, concentration changes on the biosphere-atmosphere system of West Africa. Global Change biology, 8, 1169—82.
21 Intergovernmental Panel on Climate Change (2007). Ibid.
22 Burke, M.B, Miguel, E, Satyanath, S., Dykema, J.A, and Lobell, D.A. (2009). Warming increases the risk of civil war in Africa. Proceedings of the National Academy of Sciences, 106(49), 20670–74.
23 Anderson, C.A, Bushman, B.J., and Groom, R.W (1997). Hot years and serious and deadly assault: Empirical tests of the heat hypothesis, Journal of Personality and Social Psychology, 73, 1213—23.
24 Hancock, PA, Ross, J.M., and Szalma, J.L. (2007). A meta-analysis of performance response under thermal stressors. Human Factors, 49, 851—77.
25 Burke, M.B, Mguel, E, Satyanath, S., Dykema, J.A., and Lobell, D.A. (2009). Ibid., p. 20673.
26 Bertalanffy, L. von (1969), General system theory: Foundations, development, applications. New York: G. Brazilier.
27 Cilliers, P. (1998). Complexity and postmodernism: Understanding complex systems. London: Routledge.
28 Cilliers, P. (1998). Ibid.
29 Heylighen, F., Cilliers, P, and Gershenson, C. (2005). Complexity and philosophy. Brussels, Belgium: Vrije Universiteit, Evolution, Complexity and Cognition, p. 8.
30 Canadell, J.G et al. (2007). Contributions to accelerating atmospheric CO2 growth from economic activity, carbon intensity, and efficiency of natural sinks. Proceedings of National Academy of Sciences, 104(47), 18866–70.
31 National Science Foundation (2010, March 4). Methane releases from Arctic shelf may be much larger and faster than anticipated (Press Release 10–036). Washington, DC: NSF.
32 Kune, G (2003). Anthony Eden's bile duct: Portrait of an ailing leader, ANZ Journal of Surgery, 73, 341—5. See also Pearson, J. (2003). Sir Anthony Eden and the Suez Crisis: Reluctant gamble. Basingstoke: Palgrave Macmillan.
33 Dekker, S.WA. and Hugh, T.B. (2008). Laparoscopic bile duct injury: Understanding the psychology and heuristics of the error. ANT, Journal of Surgery, 78, 1109—114.
34 The West Africa connection: How drug cartels found new routes. The Times (of London), February 28, 2009.
35 Kauffman, S. (1993). The origins of order. Oxford, UK: Oxford University Press.
36 Nyce, J.M. and Dekker, S.WA. (2010). IED casualties mask the real problem: it's us. Small Wars and Insurgencies, 21(2), 409–413.
37 Byrne, G. (2002). Flight 427: Anatomy of an air disaster. New York: Copernicus books.
38 Hutchins, E., Holder, B.E., and Alejandro Pérez, R. (2002). Culture and fight deck operations (Prepared for the Boeing Company, Sponsored Research Agreement 22—503). La Jolla, CA: University of California San Diego.
39 Onderzoeksraad voor de veiligheid (2010). Aircraft Accident Report: Neergestort tijdens de naderine, Boeing 737–800, nabij Amsterdam Schiphol Airport, 25 februari 2009. The Hague, NL: OW
40 Capra, F. (1975). The Tao of physics. London: Wildwood House, p. 139.
41 Capra, F. (1996). The web of life: A new scientific understanding of living systems. New York: Anchor Books.
42 Hollnagel, E. (2004). Barriers and accident prevention. Aldershot, UK: Ashgate Publishing Co.
43 Johnson, S. (2001). Emergence: The connected lives of ants, brains, cities and Software. New York: Scribner.
44 Leveson, N. (2006). Ibid.
45 Hollnagel, E. (2004). Ibid.
46 Weick, K.E. (1990). The vulnerable system: An analysis of the Tenerife air disaster. Journal of Management, 16, 571–593.
47 Daulerio, A,J. (2010). How one enetgy company will prevent catastrophic oil spills: Swivel-chair safety. [Online]. Available at: www.deadspin.com [accessed: 17 June 2010].
48 Glanz, J. and Rubin, A.J. (2007). From errand to fatal shot to hail of fire to 17 deaths. New York Times, October 3.
49 Risen, J. and Mazzetti, M. (2009). Blackwater guards teamed with C.I.A. on raids. International Herald Tribune, December 12—13, pp. 1 and 4.
50 Glanz, J. and Rubin, A.J. (2007, October 3). Ibid.
51 Columbia Accident Investigation (2003, August). Report Volume 1, Washington, DC: CAIB, pp. 122–3, 126 and 130.
52 Page, S.E. (2010). Diversity and complexity. Princeton, NJ: Princeton University Press.
7
Managing the Complexity of Drift
It started out as pure, clear, legitimate deals. And each deal gets a little messier and messier. We started out just taking one hit of cocaine. Next thing you know, we're importing the stuff from Colombia.
Former Enron executive
Complexity, Control and Influence
The traditional view is that organizations are Newtonian—Cartesian machines with components and linkages between them. Accidents get modeled as a sequence of events (actions-reactions) between a trigger and an outcome. But such theories can say nothing about the build-up of latent failures, about a gradual, incremental loosening or loss of control that seems to characterize system accidents and drift into failure.
Remember the basic message of this book. The growth of complexity in society has got ahead of our understanding of how complex systems work and fail. Our technologies have got ahead of our theories. Our theories are still fundamentally reductionist, componential and linear. Our technologies, however, are increasingly complex, emergent and non-linear. Or they get released into environments that make them complex, emergent and non-linear. If we keep seeing complex systems as simple systems – because of the dominant logic and inherited scientific-engineering language of Newton and Descartes – we will keep missing opportunities for better understanding of failure. We will keep missing opportunities to develop fairer responses to failure, and more effective interventions before failure.
One of the patterns through which complex systems spawn accidents is the drift into failure: a slow but steady adaptation of unruly technology to environmental pressures of scarcity and competition. Traditional safety and decision theories are incapable of dealing with drift. They only take snapshots of a system (often an already failed system, for example, layers with holes in them), and cannot bring longitudinal trajectories out in the open. They look for the engine of drift in the wrong actions or decisions of individual components. They make assumptions about rationality and human choice that are impossible to validate in complex systems.
To try to get a better handle on drift, this book has explored the ideas of complexity and systems theory. Can they help us characterize drift into failure in ways that are not componential? Can they shed light on the uncertainty and incompleteness of knowledge of the actors who work inside complex systems? Can they help us recognize the non-linearity and complexity of the intertwined social, technological and organizational processes that make a system descend into failure? Whereas previous ideas about drift and adaptation often remain Newtonian and are "systemic" only because they include more components (seeChapter 5), the ideas of complexity and system science (see Chapter 6) are opening up a new vocabulary to think about non-linear interactions, interdependencies and trajectories toward failure. The problem of drift into failure discussed in this book is not just complex in some vague metaphoric sense, but in the formal sense laid out in the Chapter 6. Drift into failure involves the interaction between diverse, interacting and adaptive entities whose micro-level behaviors produce macro-level patterns, to which they in turn adapt, creatine new patterns.
Even with this language, however, and with the generic ideas it gives us for how things might go wrong in complex systems, are we in any better position to predict and prevent failure? If we spend a lot of resources investigating past failures, does that help us at all in forestalling future ones? In a linear Newtonian system, the relationships between causes and consequences can be traced out either forward or backward in time without analytic difficulty. The conditions of a complex system, in contrast, are irreversible. A complex system because, for one, it is never static. Complex systems continually experience change as relationships and connections evolve internally and adapt to their changing environment. Results emerge in ways that cannot be traced back to the behavior of constituent components. The precise set of conditions that gave rise to this emergence is something that can never be exhaustively reconstructed. This means that the predictive power of any retrospective analysis of failure is severely limited.
Decisions in organizations, to the extent that they can be excised and described separate from context at all, are not single beads strung along some linear cause-effect sequence. Complexity argues that they are spawned and suspended in the messy interior of organizational life that influences and buffets and shapes them in a multitude of ways. Many of these ways are hard to trace retrospectively as they do, not follow documented organizational protocol but rather depend on unwritten routines, implicit expectations, professional judgments and subtle oral influences on what people deem rational or doable in any given situation.
Reconstructing events in a complex system, then, is nonsensical: the system's characteristics make it impossible. Investigations of past failures thus do not contain much predictive value for a complex system. After all, things rarely happen twice in exactly the same way, since the complex system itself is always in evolution, in flux. Even having a rough idea about how failure arises in complex systems may not help that much, as Pidgeon and O'Leary pointed out:
To understand how vulnerability to failures and accidents arises, does not automatically confer predictive knowledge to prevent future catastrophes. For in making this complex move one must forsake the familiar ground of accident analysis and disaster development to enter far more contested waters. It is no simple matter to specify how a theory of institutional vulnerability might then be transposed into one of practical resilience. Indeed, we can ask whether our analyses and theories of past accidents and disasters tell us anything useful at all for designing institutions with better future performance, or whether we are merely left with the observation that complex organizations, faced with turbulent environments, will repeatedly fail us in unpredictable ways (and that the only practical advice to risk managers is to stay fully alert to this possibility)?1
Designing our way out of drift in complex systems is probably a hopeless task indeed. We cannot design complexity, nor can we design institutions or organizations in ways that ensure that organizations don't become complex. Complexity happens even if we don't want it to happen. That doesn't mean, however, that we are only the helpless victims of Complexity. We do play an active role in shaping complexity, even if we might not know it. Speaking the language of complexity can help us find leverage points for playing that role better – or more sensitively, more attuned to the complexity of the system in which we play it.
In a complex system, an action controls almost nothing. But it influences almost everything. Most managers, just like the investigators who pick over the remains of their systems after these have crashed or burned or exploded, have been blind to this contrast. They have operated with a machine model of their world for the longest time, thinking they can control everything, or that some people at some point in time could have controlled everything (and the system is now broken because a lack of control by those individuals). This model is predicated on linear thinking, on symmetry between causes and effects, on predictability. And it is predicated on control.
The prevailing style of management in the West still reflects these notions. It was in part developed and articulated most obviously by Frederick Taylor, whose 1911 book The Principles of Scientific Management not only became a classic in the management literature, but whose effects reverberate in what managers see as common sense, as possible, to this day.
Hollowing reductionist logic to the limit, Taylor suggested we should analyse organizations down to its basic Component parts, figure out how each of them worked, develop "one best method" for how to get those parts to work and then put it back together again so as to attain the greatest possible efficiency. The workplace, the organization, a collection of humans, it was all no more than a machine, a machine that could ideally be tweaked to operate like clockwork. Ideas hardly get more purely Newtonian than that. The job of the manager was to make sure that the clock ticked without hiccups, that the machine functioned smoothly. This could be done by making assuring the quality of all its parts, and that their interaction was properly lubricated (particularly with top-down orders of what to do). The resulting organizations were very hierarchical. People were simply expected to carry out their minute jobs to a maximum efficiency, and not bother with anything outside of it. No creativity, no unpredictability. It was the embodiment of Weber's maximally rational bureaucracy. Management was about control; decisions were about Control.
