Build and deploy enough AI-infused systems and certain patterns become inescapable. After years of research and reflection on how to blend probabilistic and deterministic components, I've arrived at an observation that re-frames the entire enterprise:
In traditional software, code is the substrate. In AI systems, feedback is the substrate.
This is not a metaphor. It is a literal description of what you are working with. Code is scaffolding ~ necessary but not sufficient. The actual material you shape is the feedback loop: the cycle of output, observation, correction, and refinement that determines whether the system converges toward usefulness or drifts toward irrelevance.
This insight is scale-free. It applies whether you are crafting a single prompt, architecting a product, or structuring an organisation. What follows are three principles that emerge from taking "feedback is substrate" seriously, and a concluding question about what remains for humans when AI systems can run their own feedback loops.
Principle 1: Cultivation, Not Construction
At every scale, you are tending a process, not building an artifact.
Traditional software engineering is construction. You design, you build, you ship. The artifact ~ 'code' IS the thing. Once built, it stays built. Maintenance is repair.
AI engineering is cultivation. You plant, you tend, you prune. The artifact is not the code; it is the trajectory; the evolving relationship between the system and its environment. There is no "done." There is only "as currently calibrated."
Why does this matter? The instinct to "finish" an AI system is a category error. Stability is not a property of these systems; it is an ongoing achievement. Drift is the default state. If you are not actively cultivating, you are passively degrading.
At the interaction level, this means every prompt is a hypothesis, not a specification. You are steering toward an approximate goal, not converging on a fixed solution.
At the system level, this means building the feedback loop before building the feature. A simple system instrumented for learning will outperform a sophisticated system that is static.
At the organisational level, this means accepting that ground truth changes faster than strategy documents. The map is always slightly wrong; cultivation is how you keep it useful.
Principle 2: Legibility as Constraint on Agency
You can only safely extend autonomy as far as you can observe consequences.
There is a seductive error in AI engineering: the dream of the fully autonomous agent that handles everything without intervention. The principle of "feedback is substrate" forbids this. If feedback is what you work with, then areas without feedback are areas without material ~ as in, you cannot shape what you cannot see.
This yields what I call the Legibility Constraint: agency can only extend as far as observation. Autonomy granted beyond the boundary of legibility does not disappear; it accumulates as hidden debt. The system is doing something, but you don't know what. Eventually, the debt comes due.
Why does this matters? The question is not "how much can we automate?" but "how far can we see?" Invest in observability before investing in capability. The most dangerous systems are not the ones that fail loudly; they are the ones that drift silently.
At the interaction level, this means injecting constraints—schemas, guardrails, explicit instructions to bound the space of outputs into regions you can evaluate.
At the system level, this means building verification layers proportional to agency. Every unit of decision-making capability handed to an agent requires a corresponding investment in evaluation. This is not overhead; it is the mechanism that makes agency safe to extend.
At the organisational level, this means leaders must maintain direct contact with the technology. Intuition formed at the top degrades as it propagates downward, because the feedback is at the bottom. Observation cannot be fully delegated.
Principle 3: The Irreducibility of Tacit Knowledge
Valid knowledge in AI engineering is experiential. It can be pointed at, but not transferred.
If feedback is the substrate, then tacit knowledge is feedback that has been internalised until it no longer feels like information ~ it feels like judgment. It is the accumulated residue of a thousand micro-corrections, the intuition for when a prompt is "almost right," the sense for when a system is about to fail before the metrics show it.
This knowledge is non-fungible. It cannot be copied from blog posts, purchased from consultants, or transferred via documentation. It can only be re-derived through direct engagement with the specific problem in its specific context.
Why this aspect matters? This is the real moat. Not proprietary data, not clever architectures; but the embodied understanding that comes from having tended a particular feedback loop through its particular failures. Organisations that accumulate this knowledge build advantages that cannot be shortcut.
At the interaction level, this is why "prompt engineering, context curation & intentioanl orchestration" are real skills, yet unteachable. The principles can be articulated, but the skill lives in the fingers, in the felt sense for how a model responds to subtle variations.
At the system level, this is why the first deployment is always humbling. The gap between what you imagined and what you encounter is where the learning happens. The pain of iteration is not inefficiency; it is the only method for gathering the experiential data required.
At the organisational level, this is why hands-on technical leadership matters. The knowledge lives in the practitioners. If leadership is disconnected from practice, the organisation is navigating by a map that no longer matches the territory.
The Inversion: Deconstruct to Reconstruct
A practical consequence of these principles: you cannot graft AI onto existing workflows. The feedback loops are wrong.
Traditional workflows were designed for human cognition and human-speed iteration. The decisions are bundled, the feedback is sparse, the calibration cycles are long. Attaching AI to these workflows is like trying to cultivate a garden through a keyhole ~ you can technically do it, but you're fighting the structure.
The alternative is deconstruction. You must decompose processes to their atomic unit -- the individual decision -- and then reconstruct them with AI as the probabilistic engine and humans as the calibrators. This is uncomfortable work. It requires questioning assumptions embedded in processes that "work fine." But it is the only path to systems that genuinely leverage what AI offers.
The reconstruction pattern:
Identify the decision (not the task, rather the decision embedded in the task)
Expose it as a feedback loop (input → judgment → output → consequence → correction)
Determine what the AI can do (pattern recognition, generation, classification)
Determine what humans must do (specification, boundary-setting, reward correction)
Instrument for observation (you can only cultivate what you can see)
The Critical Question: What Do Humans Do?
The principles above assume humans in the loop. But this assumption is under pressure. Coding agents now run continuous loops, refining their outputs against reward signals until approximate success is achieved. If feedback is substrate, and agents can process feedback autonomously, what remains for humans?
The answer, I think, is that humans are what the feedback loop cannot contain.
Feedback loops operate on a specified objective, within a bounded problem space, assuming the reward signal is correct (at approximately correct). Each of these contains an irreducible human function:
Specification. Someone must decide what "good" means. Not abstractly, but in the specific context with all its contradictions. "Make the code work" is legible to an agent. "Make the code work, but maintainable, but don't break the API contract, but ship by Thursday, but don't burn out the team" ... that is judgment. The agent optimises; the human specifies what optimisation is for.
Boundary-setting. Agents operate within problem frames. But who decides where the frame is? When should the agent stop and escalate? When is a problem actually two problems? These are ontological decisions that precede and constrain any optimisation.
Reward correction. Every reward signal is a proxy for what you actually want. Proxies diverge. Goodhart's Law is not an edge case; it is the inevitable destination of any optimisation that runs long enough. Humans are what notice when the metric is being hit but the goal is being missed—when the feedback loop has started lying.
So the human function becomes: judgment under genuine ambiguity. The cases where the reward signal doesn't exist, isn't clear, or has drifted from the underlying intent.
The Organisational Implication: What & Who Survives?
If execution becomes commoditised & if agents can loop toward completion on well-specified tasks, then organisations differentiate on what precedes and surrounds the loop:
Taste. The ability to specify "good" in ways competitors cannot copy. This is why Apple was Apple. Not execution superiority, rather specification superiority. When agents can execute anything you specify, what you choose to specify becomes the competitive variable.
Problem selection. Choosing which problems to solve, and how to frame them. Agents optimise within frames; choosing the frame is upstream and higher-leverage. The organisations that identify the right problems to automate will outperform those that automate the obvious ones.
Judgment infrastructure. Not just humans-in-the-loop, but systematic ways to surface the moments where human judgment is needed. The skill is knowing when to interrupt the loop—not too early (wasted attention), not too late (compounded error).
Rich feedback substrates. The deepest advantage may simply be the quality of the feedback loops themselves. Organisations that build environments where feedback is dense, rapid, and honest will cultivate better systems—and accumulate more tacit knowledge—than those where feedback is sparse, slow, or distorted.
Summary: The Scale-Free Principles
Feedback is substrate. You are not building with code; you are cultivating with feedback. The loop is the material.
Cultivation, not construction. AI systems are tended, not built. The artifact is the trajectory. Drift is the default; stability is an ongoing achievement.
Legibility constrains agency. Autonomy can only extend as far as observation. Unobserved agency accumulates as hidden debt.
Tacit knowledge is the moat. Valid knowledge is experiential and non-fungible. It cannot be transferred—only re-derived through direct engagement.
Humans are what the loop cannot contain. Specification, boundary-setting, and reward correction remain irreducibly human. Judgment under genuine ambiguity is the function that survives.
The engineers and organisations that internalise these principles will build what others cannot copy—not because the principles are secret, but because understanding them is not the same as having lived them.
Feedback is substrate. The rest is cultivation.