Virtually all design is conducted in a state of relative ignorance of the full behaviour of the system being designed.

Henry Petroski

A State of (Relative) Ignorance

When a system is designed there is from the very beginning a need to know how it will function. Indeed, the very reason for system design is to construct an artefact that provides an intended functionality. In Henry Petroski’s book about ‘Design Paradigms,’ from which the above epigraph is taken, the topic was engineering design, and the majority of examples were physical structures and static systems, such as bridges. The ‘behaviour’ of a bridge is seemingly simple: it just has to be there and to maintain its own structure in order to allow safe passage for whoever or whatever uses it. Yet even in this case there is a ‘relative ignorance’ of what may possibly happen, as numerous examples of collapsing structures have demonstrated.
    While design may not require perfect knowledge it at least requires acceptable ignorance. Partial ignorance is unavoidable in principle as well as in practice, but it should be so little that the consequences are unnoticeable. In other words, we must be reasonably sure that the systems we build will do what they are supposed to do, that they will function reliably as intended, and that they additionally will not do anything they are not supposed to do. The latter, in the case of a bridge, means collapsing or falling down. Yet history is full of examples of just that happening from the Dee Bridge disaster (1847) to the spectacular collapse of the Tacoma Narrows in 1940 and more recently to the failure of the I-35W Mississippi river bridge (2007).

The Reasons for Ignorance
There are several reasons for this relative ignorance. One is that the properties of the materials may not be known well enough, neither how they will behave under extreme conditions (pressure, temperature, wind, etc.) nor how they will behave over time (ageing, degradation). Another is that it is uncertain what external conditions the system will be subjected to. In the case of bridges, the conditions refer to weather, changes in the chemical composition of the air, quality of maintenance, changes in ‘customer’ characteristics (e.g., heavier cars and more traffic), etc. A third reason is that work usually is done under conditions of insufficient time, information, and resources. The only thing we can know for certain is that things will happen that we have never thought of, although that in itself is of limited comfort.
    If the situation is difficult for structures and static systems, it is even worse for dynamic systems. (Note, by the way, that so far we only have referred to nominally technical systems, e.g., a bridge as a bridge. Even here, the influence of social systems is obvious, for instance in how well the technical system is maintained, how well the components meet the specifications or requirements, how well the system is designed and built, etc.) For dynamic systems it is necessary to consider also the state of the parts and components as well as the dependencies among them. Energy – and information – must be provided, mass and materials must be moved around, substances will be transformed, etc. This creates literally countless dependencies of which there is relative ignorance, but which nevertheless must work as planned in order for the system to fulfil its purpose. But as it is well nigh impossible to foresee all possible combinations, even for purely technological systems, surprises abound.

Ignorance, Risk, and Safety
Petroski’s lament can be extended from the design of technical systems to include accident investigation and risk assessment as well. Virtually all accident investigations and virtually all risk assessments are conducted in a state of relative ignorance of the full behaviour of the system being analysed – and in some cases even in a state of ignorance about the typical behaviour.
    In relation to event investigations, we can rarely if ever get all the information we need about what happened. One reason is that the search for information is influenced by biases and practical constraints. One such bias is the What-You-Look-For-Is-What-You-Find (WYLFIWYF) principle, which means that we look for what we assume is important. This precludes us from finding anything that we do not look for – serendipity excepted. Another bias is ‘illusory comprehension.’ The fact that we can squeeze events into pre-existing explanatory frameworks, all of which imply causality, means that we see causality even though it may not really be there. Among the practical constraints is the all too frequent lack of time, which means that the search for information is stopped when an acceptable explanation has been found, even though this may be incomplete or incorrect.
    In relation to risk assessment, one source of ignorance is the inescapable uncertainty about what the future will bring. An observation made by many philosophers, and often repeated by politicians, is that we cannot know with certainty what will happen in the future. The Danish philosopher Søren Kierkegaard (1813-1855) noted that while life can only be understood backwards, it must be lived forwards. Samuel Coleridge (1772-1834) somewhat more poetically noted that “the light which experience gives us is a lantern on the stern which shines only on the waves behind us.” A second source of ignorance is that most of the models or representations we use are so oversimplified that their validity is questionable. In an event tree, for instance, it is assumed that the chosen representation principle (binary branching) is an acceptable representation of reality. But that is clearly not the case, both because a distinction between fail and succeed is relative rather than absolute, and because things rarely develop in the way that was expected. A third source is the lack of imagination that partly is innate, partly comes from familiarity and habituation. A textbook example of that is Alan Greenspan’s characterisation of the 2008 financial crisis as a “once-in-a-century credit tsunami, … that … turned out to be much broader than anything I could have imagined.”

Ignorance, Complexity, and Intractability
Ignorance of the future (and to some extent also ignorance of the past) is sometimes attributed to the degree of complexity of the systems we are dealing with, or simply to the purported fact that ‘today’s systems are – or have become – complex.’ Complexity is, however, not a well-defined concept, as the following definitions exemplify:

• Mathematical complexity is a measure of the number of possible states a system can take on, when there are too many elements and relationships to be understood in simple analytic or logical ways.
• Pragmatic complexity means that a description, or a system, has many variables.
• Dynamic complexity refers to situations where cause and effect are subtle, and where the effects over time of interventions are not obvious
• Ontological complexity has no scientifically discoverable meaning, as it is impossible to refer to the complexity of a system independently of how it is described.
• Epistemological complexity can be defined as the number of parameters needed to describe a system fully in space and time. While epistemological aspects can be decomposed and interpreted recursively, ontological aspects cannot.

    It might indeed be asked whether there is a state of relative ignorance because the systems we deal with are complex, or whether we call the systems complex because we do not have – and possibly cannot have – complete knowledge about them.
    While we may entertain the hope that complete knowledge in principle is possible for technological systems (barring the vagaries of software), there is no reason for such optimism in the case of socio-technical systems. Here ignorance is a fact of life because it is impossible fully to define or describe the parameters in space or time even if we knew what they were. The main reason for this is however not that there are too many parameters, but rather that the systems are dynamic, i.e., that they continuously change.
    In order for a system to be understandable it is necessary to know what goes on ‘inside’ it, to have a sufficiently clear description or specification of the system and its functions. The same requirements must be met in order for a system to be analysed and in order for its risks to be assessed. That this must be so is obvious if we consider the opposite. If we do not have a clear description or specification of a system, and/or if we do not know what goes on ‘inside’ it, then it is clearly impossible effectively to understand it, and therefore also to investigate accidents or assess risks.
    The presence of the (relative) ignorance clashes with the assumptions of established safety analysis methods. These assumptions are a heritage from the large-scale technological systems for which the first safety assessment methods were developed in the late 1950s. Although the underlying assumptions rarely are stated explicitly, they are easy to recognize by looking at established methods, such as FMEA (Failure Mode and Effects Analysis), HAZOP (Hazard and Operability Study), Fault Trees, etc. The four main assumptions are:

• A system can be decomposed into meaningful elements (parts or typically components). Similarly, events can be decomposed into individual steps or acts. (The principle of decomposition is, of course, in conflict with the holistic principle that the whole is more than the sum of the parts.)
• Parts and components will either work or fail. In the latter case, the probability of failure can be analysed and described for each part or component individually. This is part of the rationale for focusing on the human error probability, and indeed for classifications of human errors.
• The order or sequence of events is predetermined and fixed as described by the chosen representation. If a different sequence of events needs to be considered, it is necessary to produce a new version of the representation, e.g., a new event tree or fault tree.
• Combinations of events are orderly and linear. They can be described by standard logical operators, and outputs are proportional to inputs.

    Although these assumptions may be warranted for technological systems, it is highly questionable whether they apply to social systems and organisations, or to human activities. Models and methods that require that the system in focus can be fully described will for that reason not be suitable for socio-technical systems, neither for accident analysis and nor for risk assessment. It is therefore necessary to look for methods and approaches that can be used for systems that are incompletely described or underspecified. The two types of systems can be called tractable and intractable, respectively. The differences are summarised in Table 1.1 below.