The Fundamental Insight

In 1945, when John von Neumann was informed that there were no reliable vacuum tubes and no reliable resistors for his proposed computing machine, his response was instructive: “We would have to learn how to design a reliable 40-stage machine with thousands of unreliable components” [1]. This principle, articulated at the dawn of modern computing, remains the central challenge of AI engineering today.

The difference is that we have largely forgotten it (or have remained ignorant).

Modern discourse around AI systems often proceeds as though reliability is a problem to be solved rather than a constraint to be designed around. We speak of “improving accuracy” and “reducing hallucinations” as though some asymptotic perfection awaits. But the structural regularities governing AI systems (what my colleagues and I have termed the “Physics of AI” [2]) suggest otherwise: error is inherent, interpretability is partial, and the environments in which models operate will always differ from those in which they were trained.

This is not a counsel of despair. It is an invitation to design.

The Historical Pattern

The challenge of orchestrating unreliable components into reliable systems recurs throughout the history of technology. George Dyson’s account of von Neumann’s work at the Institute for Advanced Study reveals that the engineers building ENIAC faced precisely this problem: vacuum tubes failed constantly, resistors drifted, and yet the machine had to compute reliably [1]. Their solution was not to wait for perfect components but to design redundancy and error-correction into the architecture itself. This idea persists to this day in much hardware.

This pattern repeats. In The Dream Machine, Mitchell Waldrop describes how the SAGE air-defense system addressed the same constraint through redundancy: every computer was paired with an identical backup ready to take over at a moment’s notice [3]. The components remained unreliable; the system achieved reliability through architecture.

Tracy Kidder’s The Soul of a New Machine captures the scale of this achievement in mature computing systems: “A computer like Eagle does a cycle of work in 220 billionths of a second. If it tended to fail only once every million cycles, it would be a very unreliable contraption indeed” [4]. The hardware of modern computers is remarkably reliable precisely because decades of engineering have learned to orchestrate imperfect components.

The semiconductor industry offers perhaps the most extreme example. As Chris Miller documents in Chip War, ASML’s EUV lithography tool is “the most expensive mass-produced machine tool in history, so complex it’s impossible to use without extensive training from ASML personnel, who remain on-site for the tool’s entire life span” [5]. The miracle is not simply that EUV lithography works, but that it works reliably enough to produce chips cost-effectively. ASML’s expertise, the company readily admits, was its ability to orchestrate a far-flung network of specialists whose capabilities were needed to make the dream of EUV a reality.

The Organisational Dimension

Reliability emerges not only from technical architecture but from organisational structure. Jon Gertner’s The Idea Factory describes how the great mid-century breakthroughs (the Manhattan Project, radar, the transistor) were group efforts: “a compilation of the ideas and inventions of individuals bound together with common purposes and complementary talents” [6]. The phone system, with its almost unfathomable complexity, was by definition a group effort.

Ben Rich’s account of the Skunk Works reveals a specific organisational form optimised for this challenge: “a small, intensely cohesive group consisting of about fifty veteran engineers and designers and a hundred or so expert machinists and shop workers” [7]. The Skunk Works succeeded by staying free from bureaucratic interference, maintaining the tight feedback loops necessary to catch and correct errors before they propagated.

G. Pascal Zachary’s Showstopper, chronicling the development of Windows NT, identifies the tension directly: “Writing good code is increasingly the work of large teams. Yet such teams often sink into mediocrity because their size alone can breed bureaucracy and sterility. The challenge for every large team is to organise its diverse talents while encouraging leadership and flexibility” [8]. The unreliable elements here include not just code but people, processes, and the interfaces between them.

Failure as Teacher

If unreliability is inherent, then failure becomes a source of information rather than merely a cost to be avoided. Henry Petroski articulates this principle in To Engineer Is Human: “Sometimes mistakes occur. Then it is failure analysis, the discipline that seeks to reassemble the whole into something greater than the sum of its broken parts, provides the engineer with caveats for future designs. Ironically, structural failure and not success improves the safety of later generations of a design” [9].

Don Norman extends this insight to system design in The Design of Everyday Things, introducing the Swiss cheese model: “Design redundancy and layers of defense... The metaphor illustrates the futility of trying to find the one underlying cause of an accident (usually some person) and punishing the culprit. Instead, we need to think about systems, about all the interacting factors that lead to human error and then to accidents, and devise ways to make the systems, as a whole, more reliable” [10].

These historical and theoretical observations frame what I have learned empirically across thirty years of building systems that must function despite the unreliability of their components.

My long time mentor/collaborator and Director of Deakin Applied AI Initiative, Prof Kon Mouzakis presented the direct value of failure in a talk; in that, “failures forces reflection far more than the wins & to never shy away from that reflection as that is the way to learn and improve”.

A Personal Archaeology of Unreliability

My own encounters with this principle began long before the current AI moment, in domains that would not have been called “artificial intelligence” at all.

1995: Natural Language as Approximation

In a manufacturing facility in 1995, I built what would now be called an NLP interface: a system that allowed operators to extract defect reports and process summaries using natural language queries. The challenge was immediate. Human language is underspecified. The same question can be asked in hundreds of ways. The same words can mean different things depending on context.

My solution was not to make the parser “smarter” but to acknowledge its limits (the truth is, I also did not have any knowledge of how to build a ‘smart’ algorithm; I barely knew enough about how to make reliable parsers). I manually identified hundreds of query permutations and mapped them to a smaller set of canonical report types (this was done by talking to people on the manufacturing line in person over months). When an operator’s query did not match any known pattern, an algorithm approximated the nearest plausible report (it was not smart as far as I can remember; it was error-corrected into existence is the best I can say). The system was, in a precise sense, unreliable: it could not guarantee correct interpretation of arbitrary input. But it was useful because it collapsed an unbounded input space into a bounded output space where errors could be caught and corrected reasonably quickly.

The parsers and mappers evolved manually over time. Humans remained in the loop (that is mostly me), not as a failure mode, but as the substrate on which meaning ultimately rested. This glued up system, surprisingly, lasted a good number of years till the manufacturing plant itself closed. The main learning for me was that at a system level, you can achieve much more reliability than is feasible at the individual component level.

Cardboard and Constraints

Later, I worked on control systems for cardboard manufacturing: the algorithms that determined how boxes would be cut, folded, and printed. Here the unreliability was physical. Printing speeds varied. Color calibrations drifted. The machines had limit functions beyond which they could not operate.

The humans operating these machines also brought their own uniquely human & structural traits: shifts changed, expertise varied. The solution was to orchestrate human judgment with deterministic constraint optimisation. Critically, decisions had to made when to take the human-out-of-loop. The computer held the hard boundaries (what was physically possible); the human made decisions within those boundaries (what was desirable). Neither alone was sufficient. This mirrors what Gertner observed at Bell Labs: the phone system’s complexity required “a vast amount of multidisciplinary expertise... to bring any given project to fruition” [6].

Telecommunications: When Software Cannot Save You

In telecommunications systems, I encountered a different species of unreliability: the failure modes of distributed systems. Hardware failed. Software failed. Networks partitioned. The crucial insight was that these failures could not be fixed in software.

There was no algorithm that would make a failed switch unfail.

What could be done was detection and response. We built monitoring systems using tactics that would now be called ‘metamorphic testing’: checking constraints that should always hold, and flagging when they did not. Simple logistic regressions determined which anomalies merited human attention. The triggers were statistical; the responses were human. The unreliability was not eliminated but made legible. This is Norman’s Swiss cheese principle in action: layers of defence, each imperfect, combining to catch what the others miss [10].

HR Systems: Defaults as Design

A human resources ranking system taught me about a subtler form of unreliability: the gap between what a system does and what users believe it does. We integrated rule engines with statistical estimation to rank and sort job candidates. The statistical component was explicitly probabilistic: it offered rankings, not verdicts.

But the GUI masked this uncertainty. Information was “all presented” (technically true), but the defaults shaped behavior in ways users did not fully appreciate. When the volume of candidates exceeded the time available to review them, the default sort order became, in effect, the decision. The unreliability of the statistics was hidden by the UX layer, which gave the impression of comprehensiveness.

This was not deception exactly, but it was design that obscured the underlying non-determinism. I learned that such masking could be useful (reducing cognitive load) or dangerous (inducing false confidence), depending on whether users understood what they were not seeing.

Computer Vision: Humans Augmented to Their Limit

The domain that most directly prefigured current AI challenges was computer vision. In the early days of automated BIB detection for marathon runners, accuracy hovered between 85% and 95%. This sounds impressive until you multiply by tens of thousands of runners. At scale, even 5% error produces thousands of misidentifications.

The solution was not to wait for better models but to design around the error rate. Manual data checking layers were created. The AI did not replace humans; it augmented them, speeding their work up to a limit determined by human attention and the error rate of the system. Similar patterns emerged in photo analysis: the statistical layer proposed, the human layer disposed.

Our subsequent research on computer vision services revealed that this unreliability extends to the cloud services developers increasingly depend upon. We found that these services evolve in unpredictable ways, provide no notification to developers, and changes are undocumented [11]. Labels and confidence scores shift over time as underlying models are retrained, creating maintenance challenges for any application built on top of them. Intelligent services, we concluded, are not “plug-and-play”: developers are locked into whichever vendor they choose because there is inherent inconsistency between services in both the vocabulary and ontologies they use [11].

This led us to propose architectural tactics for guarding against such evolution [12], including proxy servers that monitor for behavioural drift and threshold-tuning tools that help developers calibrate their systems to specific domains [13]. The underlying principle in all this work is the same: accept unreliability as given, design systems that detect and respond to it.

X-rays and Cognitive Load

Working with medical imaging (X-rays specifically), the focus shifted from raw accuracy to interface design. The question became: how do we present AI-assisted analysis in ways that augment rather than overwhelm human cognition? The unreliability of the AI was a given; the challenge was to expose the right information at the right moment, neither hiding uncertainty nor drowning clinicians in caveats.

This work revealed that “accuracy” is not a single number but a distribution across contexts, and that the usefulness of a system depends as much on how it fails as on how it succeeds.

Software Evolution: Homeostatic Envelopes

My research on software evolution revealed that not all couplings between components remain viable under evolutionary pressure. Systems have what might be called homeostatic envelopes: ranges of parameter variation within which they remain stable. Push beyond these ranges and the system degrades or fails.

This applies to both deterministic and probabilistic systems. A well-tested function can break when its dependencies change. A well-calibrated model can degrade when its input distribution shifts. The lesson is that reliability is not a property of components but of systems, and systems exist in time. Petroski’s observation about structural engineering applies equally here: failure analysis improves the safety of later generations of a design [9].

Smart Homes and Action Spaces

When we placed sensors in smart homes for the elderly, the full weight of these lessons became clear. The sensors had innate error rates. The inference algorithms (estimating what was happening in the home from sensor readings) were probabilistic. The space of possible internal states was vast.

But the space of available actions was small: alert a caregiver, adjust a thermostat, log an anomaly for later review. We had to collapse the internal complexity into this smaller action space. The details of the probabilistic inference were not exposed to end users because they were not actionable. What mattered was whether the system’s outputs fell within the set of responses the care network could actually provide.

I noticed this pattern again in a COVID-era triaging system: vast data, complex inference, but ultimately a small set of available interventions. And again in recent work on network security: the data space dwarfs the decision space, which dwarfs the action space.

The Recurring Pattern

Across thirty years and a dozen domains, the same structural pattern recurs:

• Unreliability is inherent at every layer: sensors, algorithms, humans, networks, organisations.

• The data space is enormous; the decision space is smaller; the action space is smaller still ~ this shrinking is actually helpful as mostly, humans care about valid and useful actions/interventions.

• Reliability emerges from orchestration, not from perfecting components: human layers, monitoring systems, constraint optimisation, threshold-based triggers, UX design that shapes attention.

• Masking uncertainty can be useful or dangerous, depending on whether the system’s failure modes align with the user’s mental model.

• Systems exist in time: what works today may not work tomorrow as environments, dependencies, and expectations evolve.

This is the same pattern that Dyson documents in von Neumann’s lab, that Kidder captures in the Eagle project, that Rich describes at Skunk Works, that Miller traces through the semiconductor supply chain. The scale and domain vary; the fundamental challenge does not.

The Physics of AI

These observations have led my colleagues and me to propose what we call the “Physics of AI” [2]: a framework that articulates the structural constraints governing AI systems, analogous to how physical laws govern motion and energy.

We identify nine such constraints, grouped into three categories:

Adaptation Constraints address how AI systems learn and change:

• Error is inherent: all models approximate reality; uncertainty cannot be eliminated.

• Interpretability is partial: explanations are simplifications of what happens inside models.

• Training ≠ Deployment: real-world environments differ from training conditions.

Regulation Constraints address how AI behavior is bounded and aligned:

• Human feedback is substrate: AI systems do not define their own objectives; humans do.

• Resource boundaries: quotas, latency, and cost constrain what architectures are feasible.

• Partial alignment: quantitative metrics capture only fragments of intent.

• Robustness fragility: small changes can cause large behavioral shifts.

Resilience Constraints address the system’s capacity to withstand shocks:

• Entanglement: data, code, and humans form interdependent loops where changing anything changes everything.

• Benchmark expiry: static benchmarks decay as tasks, data, and expectations evolve.

These are not optional design choices but structural features that any engineering approach must design around. They are the reason that building reliable AI systems from unreliable AI components requires the same conceptual move von Neumann made in 1945: accepting imperfection as a constraint and designing coherence at the system level.

From Principles to Practices

What does this mean for practice? Our empirical work across multiple case studies suggests several implications [2]:

Invest in experiment tracking infrastructure. If error is inherent and environments drift, then every deployment is an experiment. The infrastructure to track, compare, and learn from experiments is not overhead; it is core capability.

Make logging and observability defaults, not afterthoughts. If interpretability is partial and robustness is fragile, then the ability to see what is happening in a system is essential for maintaining it.

Foster learning-centered experimentation cultures. If many experiments will “fail” (produce results that do not meet initial hopes), then organizations must structure themselves to learn from these outcomes rather than punish them. This is the Skunk Works insight: small, cohesive teams with tight feedback loops outperform bureaucratic structures when the work involves navigating uncertainty [7].

Define business metrics for the problem. If alignment is partial and benchmarks expire, then the only durable measure of success is whether the system produces value for the humans it serves. Technical metrics are proxies; human outcomes are the thing itself.

Know the bias of your evaluators. If human feedback is substrate, then the humans providing feedback shape the system. Understanding their biases (and the biases of any AI evaluators) is essential for calibration.

The Silver Lining

There is a silver lining in this framework.

The Physics of AI constrains engineering methodology and practices, but it does not necessarily constrain outcomes. The goal is not to eliminate unreliability (impossible) but to orchestrate unreliable parts into reliable wholes (achievable, with effort).

This is what the builders of ENIAC learned [1]. It is what the SAGE engineers learned [3]. It is what every semiconductor fabrication facility learns anew [5]. It is what the telephone system required [6]. The field of AI engineering is reflecting and integrating these lessons too.

The question is not whether our AI components will be reliable. They will not be. The question is whether we can design systems that acknowledge this fact and achieve useful coherence anyway.

Von Neumann’s answer, in 1945, was yes. Ours, in 2026, must be the same.

This essay draws on applied research work conducted with colleagues at the Deakin Applied Artificial Intelligence Initiative, Deakin University. The "Physics of AI" framework appears in our paper at CAIN 2026 (co-located with ICSE 2026).

References

[1] G. Dyson, Turing’s Cathedral: The Origins of the Digital Universe. New York: Pantheon Books, 2012.

[2] S. Barnett, A. Pasquini, S. Kurniawan, S. Sivasothy, R. Hill, and R. Vasa, “The Physics of AI,” in Proc. 5th IEEE/ACM International Conference on AI Engineering - Software Engineering for AI (CAIN ‘26), Rio de Janeiro, Brazil, Apr. 2026.

[3] M. M. Waldrop, The Dream Machine: J.C.R. Licklider and the Revolution That Made Computing Personal. New York: Viking, 2001.

[4] T. Kidder, The Soul of a New Machine. Boston: Little, Brown and Company, 1981.

[5] C. Miller, Chip War: The Fight for the World’s Most Critical Technology. New York: Scribner, 2022.

[6] J. Gertner, The Idea Factory: Bell Labs and the Great Age of American Innovation. New York: Penguin Press, 2012.

[7] B. R. Rich and L. Janos, Skunk Works: A Personal Memoir of My Years at Lockheed. Boston: Little, Brown and Company, 1994.

[8] G. P. Zachary, Showstopper! The Breakneck Race to Create Windows NT and the Next Generation at Microsoft. New York: Free Press, 1994.

[9] H. Petroski, To Engineer Is Human: The Role of Failure in Successful Design. New York: St. Martin’s Press, 1985.

[10] D. A. Norman, The Design of Everyday Things, Revised and Expanded Edition. New York: Basic Books, 2013.

[11] A. Cummaudo, R. Vasa, J. Grundy, M. Abdelrazek, and A. Cain, “Losing Confidence in Quality: Unspoken Evolution of Computer Vision Services,” in Proc. 35th IEEE International Conference on Software Maintenance and Evolution (ICSME ‘19), Cleveland, OH, USA, 2019, pp. 333–342.

[12] A. Cummaudo, S. Barnett, R. Vasa, J. Grundy, and M. Abdelrazek, “Beware the Evolving ‘Intelligent’ Web Service! An Integration Architecture Tactic to Guard AI-First Components,” in Proc. 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE ‘20), Virtual Event, USA, 2020, pp. 269–280.

[13] A. Cummaudo, S. Barnett, R. Vasa, and J. Grundy, “Threshy: Supporting Safe Usage of Intelligent Web Services,” in Proc. 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE ‘20), Virtual Event, USA, 2020, pp. 1645–1649.