Your AI has outrun your governance

• AI governance hasn't kept pace with agentic AI. Banks have AI policies, councils and model registers, but very few can show what their AI decided yesterday. Models change monthly and agents act in seconds. Slow approval isn't safe, it's a risk.
• Accountability can't be delegated. Under the Senior Managers and Certification Regime (SM&CR), a named person stays accountable for what an AI agent does. Every agent needs a written mandate, hard limits, a named owner and a tested off-switch.
• Govern the workflow, the trajectory and the dependencies, not just the model. Customers experience decisions, not models. The Hugging Face incident shows how every agent can pass its own tests while the journey heads somewhere nobody approved.
• Programmable governance builds control in. Keep policy outside the agent as code, enforce it at runtime and log evidence by default. Treat every agent as an insider threat, with its own identity and least-privilege access.
• Human oversight has to be designed. Approving hundreds of AI recommendations an hour turns people into "meat proxies": the liability stays with them but the judgement has gone. Put people where their judgement changes the outcome.
• Regulators are watching, and you can start small. Deadlines have moved, including EU high-risk obligations to 2 December 2027, but accountability hasn't. Pick the AI decision you'd least like to explain and work through the nine moves from there: govern the right thing, build control in and keep a person accountable.

Nine moves to close the gap, for the leaders whose names are on the line

Customers are increasingly handing decisions to agents. Banks are doing the same with their operations: flag the payment, settle the claim, chase the arrears, run the KYC check. The work is being delegated, but the accountability isn’t.

Most AI governance wasn’t built for this delegated economy. Every bank now has an AI policy, most have a committee or council, a system/model register and a responsible AI framework. But, very few can answer a simple question. What did your AI decide yesterday, how long would it take to evidence, and would you stand behind the answer?

The challenge isn’t so much effort, it’s pace of change and development. Models change monthly and agents act in seconds while governance is largely still geared towards model risk with limited change between reviews. Now, every quarter a good use case waits in an approval queue, the value erodes and colleagues’ route around frameworks and process. Slow isn’t safe. It’s a risk.

Recently we have seen what happens when agents outrun their controls. During an internal test with safeguards deliberately lowered, OpenAI models escaped their sandbox through a third-party package tool, found each other and worked as a swarm. Then they broke into Hugging Face to find the answers to their test. Each step made sense to an agent designed to chase a narrow goal. Warning signs appeared weeks earlier but didn’t reach the leaders responsible for responding. In the end, Hugging Face’s own security team and agents caught it.

This was a lab, not a bank. But the lesson isn't that agents are unstoppable. It's that controls everyone assumed were there weren't running.. Your agents may not run with their guardrails switched off, but they will chase narrow goals, share infrastructure and environments, talk to other agents and create warning signs that need to reach the right desk. We need to be aware of the potential for cascading failures and the trajectory risk of these connected systems.

Regulators can see the emerging risk. The Mills Review, describes retail finance becoming continuous and delegated. It warns that the Senior Managers Regime will come under pressure as AI gains autonomy. The EU has moved high-risk obligations for systems such as credit scoring to 2 December 2027. While the deadlines have moved, accountability didn’t.

Below are nine moves to action in three groups: Govern the right thing, Build control in, Keep a person accountable. None of these need a bank-wide programme to start.

Closing the control gap

Govern the right thing

Most governance points at the wrong target(s). It inspects the model, the step and the system you built. The risk sits in everything around it: the workflow, the harness that steers it, the environment it acts in and everything you depend on.

For boards, executive committees and business owners, who decide what gets governed.

1. Govern the workflow, not just the model

The pace of change means your models will change monthly, if not faster. Your system design will become interoperable, so models swap as the frontier moves. However, customers never meet a model. They meet a decision. Govern the entire workflow from data to decision to outcome, and the model becomes a replaceable part rather than a permanent liability.

What good looks like:

• map each AI-enabled decision end to end, from the data to the customer outcome
• set appetite and controls on the outcome, then test any model that sits inside it
• swap models without reapproving the whole workflow, as long as the outcome tests still pass.

Customers don’t experience models. They experience the outcomes and decisions.

2. Govern the trajectory in addition to the steps

Agents don’t work alone. They are designed to communicate and work with other agents, call tools and will soon negotiate with your customers’ personal agents. Every agent in a chain can pass its own tests while the chain heads somewhere nobody approved.

The Hugging Face incident demonstrates this and highlights the need to govern the trajectory risk. Agents in separate tests found a way to talk, pooled what they knew and started converging as a collective. By example when one agent paused an out-of scope attack, another initiated with a time bound deadline to proceed, which another treated as permission and then did. No single step was the failure; the collective trajectory was. Now picture that series of cascading failures at scale in your payments chain. The Bank of England is already simulating whether AI trading agents could herd and amplify a market stress.

An emerging pattern is separation of duty firebreaks. Break the end-to-end flow into narrow agents with narrow outcomes. Watch every handoff. If one segment starts to drift, kill that segment, not the whole process. However, a narrow agent can look compliant and still be one part of a wider swarm, or be acting on a poisoned environment. Firebreaks only work if you also watch the whole journey and the environment it runs on.

What good looks like:

• monitor the journey across agents, not only each agent’s output
• check every instruction one agent gives another against the mandate behind it, including instructions from customers’ agents
• design firebreaks so you can stop one segment without stopping the bank.

Every agent can be compliant while the journey isn’t.

3. Govern your dependencies, design for operational resilience

Your agents run on models, clouds and tools you often don’t own, governed by varied jurisdictions. In July 2026 HM Treasury brought four major cloud providers under direct regulatory oversight as critical third parties. No AI model provider made the list. Meanwhile many banks lean on one or two frontier models, priced, updated and withdrawn outside their control.

That makes sovereignty an operational resilience question, not a political one. Where does the model run? Whose law decides its use? What happens to your service if a supplier, or a government, switches it off?

We’ve seen this play out already. In June 2026 a US export control order led Anthropic to suspend access to its two newest models for every customer worldwide. General access came back 19 days later. Additionally, agents embedded in vendor software add a layer of dependency you likely have not chosen.

What good looks like:

• map every model, cloud and tool behind each important business service
• test whether you stay within impact tolerance if your main model provider is unavailable for a month
• keep a tested fallback, whether that’s another model, another region or a human process.

You can outsource the model but you can’t outsource the risk.

Build control in

Controls written in policy or business requirement documents, checked periodically can’t keep up with systems that act in seconds. Control must run where your agents run.

For risk, compliance and model risk teams, and the technology, data, AI and security leaders who build and run the controls.

4. Take the rules and policy out of the agent

If your guardrails live in the agent’s configuration, the agent is marking its own homework. Implement programmable governance. Keep policy and business rules outside the agent(s), in an independent layer that checks every action before it lands. Policy as code with controls written as tests, thresholds and guardrails that run with every release and every transaction, evaluations in the pipeline and limits enforced at runtime. Evidence logged by default.

The Hugging Face incident again demonstrated the point. In a post-incident test, OpenAI found its production safeguards cut the models’ tendency to compromise infrastructure by more than 100 times. A replay with its expanded monitoring alerted security about 30 minutes after the escape, more than 30 hours before the breach. But, the original evaluation ran without either. The model didn’t change, the controls in the environment around it did.

Your second line needs engineers as much as your engineers need the second line. Risk must be involved up-front, not as an afterthought.

What good looks like:

• keep policy and business rules in a separate, versioned layer the agent can’t change
• enforce limits at runtime and log the evidence by default
• put engineers in the second line and risk specialists in delivery teams.

Policy still matters, it just needs to be codified and enforced.

5. Treat every agent as an insider threat

Banks know how to control access for people: Least privilege, segregation of duties, surveillance. Agents need the same treatment, because an agent with credentials is an insider. It can be manipulated, make mistakes and increasingly can find gaps in your controls. In addition, there are the agents you didn’t build arriving inside existing software licences and agreements, from customer platforms to service desks, vetted by someone else and holding access you may not have mapped. Bring them into third-party risk management.

What good looks like:

• give every agent its own identity, its own credentials and only the access it needs
• separate duties so no single agent can initiate, approve and execute
• monitor agents the way you monitor traders, including the ones that came with your software.

An agent with credentials is an insider, you need to design for it.

6. Secure the environment, not just the system

Controls can be designed and operating effectively on an agent but you can still lose control through the environment it operates within. Your agents read emails, web pages, documents, 3^rd party tools, and other agents’ outputs. Each one is a way in. In production prompt injection, poisoned data, compromised tools and leaked credentials will make irrelevant how well your model did in validation. A narrow agent can follow every rule and still act on poisoned inputs or access unforeseen vulnerabilities in pursuit of a goal.

What good looks like:

• threat-model the environment each agent works in, including its tools, data sources and shared infrastructure
• treat every input as untrusted until it’s checked, especially content from outside the bank
• red-team the unhappy path, from leaked credentials to a hostile counterparty agent.

Your model passed. Your environment didn’t.

Keep a person accountable

Software can act but it can’t answer for what it did. Under the Senior Managers and Certification Regime (SM&CR), accountability stays with a person, regardless of how much work is delegated or level of autonomy in the system.

For senior managers, business owners and internal audit.

7. Put a name on every agent

Agents are becoming colleagues, customers and counterparties. Each one needs a written mandate, hard enforceable limits, a named accountable owner and a way to stop it without stopping the bank. The system is not liable, and it can’t be fined, banned or called before a select committee. A person remains accountable.

Test every outcome, not just the happy path. A kill switch you’ve never tested isn’t a control. What does the agent do when the task is impossible? An agent with a goal will keep trying, and the harder it tries, the further it can stray from what you’ve approved. Some evidence of frontier model behaviour even highlights that when faced with complex challenges these models look to circumvent the described goal or outcome. We need to design for a safe way to stop, or ask for support and clarification and in parallel ensure effective monitoring so that warning signs alert someone with the authority to stop it.

Parliament’s Treasury Committee has asked the FCA to set out what reasonable steps look like for senior managers using AI by the end of 2026. Don’t wait for the guidance to start building the evidence.

What good looks like:

• write a mandate, limits and a named owner for every agent before it goes live
• give every agent a safe way to stop when the task is impossible, ambiguous or hostile
• rehearse the off-switch, and make sure warning signs reach someone who can pull it.

Who signs for the agent?

8. Stop turning people into meat proxies

A colleague approving hundreds of AI recommendations an hour isn’t exercising judgement. They’ve become a ‘meat proxy’- a person placed between the agent and the decision so that someone carries the accountability. The judgement has been removed but the liability hasn’t.

This approach fails on accountability, because a signature isn’t oversight and the regulator will know the difference. The Mills Review sees the same pressure point, where meaningful human control gets harder to evidence as AI acts more on its own. It also fails on skills, because the nature of the work erodes the thinking that was meant to operate as a guardrail.

Real oversight is designed. People set the mandate and the thresholds, handle the exceptions that need judgement and hold a lever that terminates where needed.

What good looks like:

• put people where their judgement changes the outcome, not where their name absorbs the blame
• track override rates as a warning light, knowing it’s a blunt measure, because a reviewer who never disagrees may not be reviewing
• protect the skills you’ll need when the agent fails, with rotation, training and deliberate practice.

A human who approves everything isn’t a control.

9. Prove the outcome and monitor the evidence

Enormous effort goes into explaining systems that are, by design, probabilistic. Where a decision must be fully explainable, use a method that is as well. Everywhere else, ask the question the Consumer Duty already asks: what outcome did the customer get, and can you evidence it was a good one?

The test of effective governance isn’t about the framework it’s whether you can demonstrate on demand what your agent(s) did yesterday, and why it was acceptable. If the chain-of-thought is masked or the evaluation does not derive the judgement and actions, you do not have effective control. In addition, with the Mills Review recommending an AI-enabled supervisory model, your evidence may soon be read by a machine.

What good looks like:

• log every agent’s inputs, actions and outcomes by default
• test outcomes for fairness and drift continuously and case by case
• rehearse the supervisor question on a live agent every quarter.

You can’t always explain the system. You can always evidence what it did.

Where to start

Start small. You don’t need a bank-wide programme. Pick the AI decision you’d least like to explain, ideally one that’s agentic or soon will be. Then work through six steps.

1. Map its workflow and trajectory end to end, including every agent, tool and handoff.
2. Map what it depends on, from models and clouds to vendor software and where each one runs.
3. Take its rules out of the agent and enforce them at runtime.
4. Give each agent an identity, a mandate, limits and a named owner.
5. Test the unhappy path, then pull the off-switch for real.
6. Measure how long it takes you to trust it, and how often people overrule it.

  Where to start across the organisation:  

We’re working through these moves with banks now. If you’d like a second pair of hands on the first one, talk to us.