Every Intervention Is Also an Experiment

If employees are not using a new AI assistant, more training can sound like an obvious response.

But that recommendation already contains a conclusion about the problem. It assumes that people would use the assistant if they understood it better.

Perhaps they do understand it. Perhaps the output needs too much correction, the source information is incomplete or using the assistant creates an awkward extra step. Training will not necessarily change any of those conditions.

The proposed solution is a hypothesis about how the organisation works. So is a process redesign, a new review rule or a decision to delegate more to an agent.

We should design changes in a way that makes those hypotheses visible—and gives us a useful answer when we test them.

The mechanism matters

A business outcome can be ambitious while the proposed mechanism remains vague.

“Increase customer capacity through AI” states a direction. It does not explain what will happen in the operation.

A more useful proposition might be that giving a preparation agent better customer context will reduce the time people spend correcting its work, allowing them to spend more time resolving customer issues.

That explanation identifies a chain that can be examined. Context improves. Correction work falls. Useful capacity becomes available. The team uses that capacity for the intended outcome.

If any part of the chain fails, the result should affect the next decision. The agent might still produce poor responses. The correction effort might fall without affecting overall workload. Another bottleneck might prevent the team from using the time differently.

Knowing which part failed is much more useful than deciding that “the pilot did not work”.

A change can teach as well as improve

There is a useful idea in adaptive control: an action can affect the system and also reveal information about it. Dual control studies that tension—acting for a result while learning about the system that will determine later results. [1]

For organisational change, this suggests that the largest estimated immediate benefit is not always the best next move.

Imagine a service function considering a wider agent rollout. It is uncertain whether the constraint is data quality, employee capability or the way work is handed over.

A bounded intervention could hold the agent and task broadly consistent while improving the context supplied in one workflow. The team could examine correction work, completion, quality and what people did with any capacity released.

If the result helps distinguish an information problem from a capability problem, it can improve a much larger investment decision. The value of the intervention includes what it teaches.

Active inference offers a related theoretical distinction between actions that help realise preferred outcomes and actions that reduce uncertainty. [2] The useful management question is plain: which change is likely to help, and what will we understand better after making it?

This does not turn employees into experimental subjects without a say. Legitimate purpose, participation, quality controls and appropriate authority belong in the design from the beginning.

Design the learning before the launch

A pilot needs a question sharp enough to receive an inconvenient answer.

Before starting, the team should describe the current condition, the proposed mechanism and the result it expects. It should identify whose work will be affected, what would count as meaningful improvement and what must not get worse.

Agree when a meaningful signal should appear, when the result will be reviewed and who will decide whether to continue, revise or stop.

It should also decide what would weaken the recommendation.

If better context is supposed to reduce correction work, what would it mean if correction remained unchanged? If the change increases output but creates more exceptions downstream, is that acceptable? If it works only for experienced staff, what does that imply about wider rollout?

These questions prevent every result being absorbed into the same conclusion: more time, more scale or more training is needed.

They also improve the choice of measures. Logins and satisfaction can be useful, but the evidence should reach the proposed mechanism. A claim about capacity needs evidence about work and its destination. A claim about better service needs evidence about the customer result.

Establish what actually reached the work

A launch date is not a description of exposure.

The system may have been available only to part of the team. People may have used different versions. Some may have retained the old workflow. A new control may have been formally introduced but rarely applied.

Before interpreting the outcome, establish what was actually put into practice.

That might involve confirming which tasks reached the agent, what context it received, where human review occurred and whether the intended sequence was followed. It also means noticing other changes that could affect the result.

Without this, a company can reject a useful idea that was never properly tried, or credit an intervention for an improvement that happened elsewhere.

Exposure is an unglamorous detail with major consequences. It connects the change described in the plan to the work people actually performed.

Decide how much confidence the result deserves

A business improves after an intervention. How much of that improvement should be attributed to it?

Demand may have changed. Another system may have launched. The people most willing to try the new method may differ from those who did not. An outcome may reflect several changes acting together.

Causal inference is concerned with reasoning about what would have happened under a different action, not merely describing what followed the chosen one. Pearl’s work makes the role of explicit causal assumptions clear. [3]

The practical implication is to match confidence to evidence.

A before-and-after comparison can show that something moved. Evidence about timing, exposure and the expected mechanism can strengthen the case that the intervention contributed. A credible comparison or well-designed experiment may support a stronger conclusion.

The name of the method is not enough. A comparison group must be relevant, and a staggered rollout must account for other changes over time. The assumptions matter.

For many operating decisions, perfect attribution will be unavailable. Leaders can still act on proportionate evidence while keeping uncertainty visible. The requirement is to avoid converting a plausible contribution into a certainty that later decisions cannot safely rely on.

Make the first move proportionate

The size of a change should reflect both its potential and how much is understood.

When the mechanism is familiar, the consequences are manageable and the evidence is strong, a wider intervention may be sensible. When uncertainty is high, a smaller, reversible move may be worth more.

Reversibility has to be real. Can the earlier workflow be restored? Can people recover the information they need? Who can pause the change if a quality or control problem appears? What would make the team stop?

A small pilot can still be poorly designed. Choosing only enthusiastic users, excluding difficult cases or measuring the easiest success indicator can produce confidence without useful learning.

The better test includes enough of the work’s real variation to inform the decision it is supposed to support. Its scope is bounded, but its question is consequential.

Let the answer change the plan

Learning is visible when a result changes what the company does.

If improved source information helps and extra training does not, the next investment should reflect that. If a control is protecting quality, it should be retained or redesigned with care. If an intervention shifts work elsewhere, the next review should include that part of the operation.

A failed hypothesis can be valuable when it prevents a larger mistaken commitment. That does not make every failed project a learning success. The insight has to be credible, recoverable and used.

This is the appeal of closed-loop transformation. Each change becomes an opportunity to improve both the operation and the understanding behind the next decision.

The self-improving company is willing to be corrected by what happens in its own work. It makes that correction early enough to matter.

References

[1] Filatov, N. M. & Unbehauen, H. (2004). Adaptive Dual Control: Theory and Applications. Springer, Lecture Notes in Control and Information Sciences, 302. https://link.springer.com/book/10.1007/b96083

[2] Da Costa, L., Parr, T., Sajid, N., Veselic, S., Neacsu, V. & Friston, K. (2020). Active inference on discrete state-spaces: A synthesis. Journal of Mathematical Psychology, 99, 102447. https://discovery.ucl.ac.uk/id/eprint/10115398/

[3] Pearl, J. (1995). Causal diagrams for empirical research. Biometrika, 82(4), 669–688. https://ftp.cs.ucla.edu/pub/stat_ser/R218-B-L.pdf