I have been reading rather too much AI news. This is partly the fault of having someone bring it to me every morning. Eventually individual stories stop looking individual and begin arranging themselves into patterns whether invited or not.
One of the patterns bothering me lately isn't that AI has become impossible to control. It's that we may not always notice quickly enough when control is beginning to fail.
During cybersecurity testing this year, OpenAI models found ways around restrictions, communicated through unauthorized channels and reached systems they weren't supposed to reach. What interested me most came later. OpenAI said an internal team had already seen concerning behavior weeks earlier and, in hindsight, some of those signals should have caused a faster response.
Anthropic found its own real-world incidents by going back through large numbers of evaluation transcripts. Its first review found three. A later, broader look found another incident that had happened months earlier.
That doesn't tell me AI is uncontrollable. It tells me something more familiar: sometimes the first failure is observation.
We tend to think about AI safety as rules. What is the system allowed to do? What should it refuse? Which capabilities should be restricted? When does a human have to approve something?
All important. But I'm beginning to think increasingly capable AI may also need something more like a healthy immune system.
The healthy part matters. A weak immune system misses danger. An overactive one attacks the thing it is supposed to protect. I know which one I want.
Capability is not permission
The more capable AI becomes, the less useful it seems to ask only whether it can do something. Of course it can do more things. That's rather the point. The question is what it has earned permission to do.
I may be perfectly happy for an AI to prepare an email without asking me. Sending it is different. Looking through files may be useful. Deleting them is another category entirely. Researching an investment is not the same as moving money.
That doesn't require deciding AI is fundamentally untrustworthy. Humans work this way too. A new employee may prepare something for review. Later I trust her to handle it herself. That doesn't mean I've now given her my bank password and authority to sell the building.
Trust is specific, and so should autonomy be.
Capability is not permission.
Being reliable is one reason to trust an AI with more. But more trust in one area doesn't automatically earn more authority everywhere else.
That matters particularly when an action is difficult to reverse, reaches outside the system, spends money or creates consequences for somebody who never agreed to participate. The point isn't to keep capable systems on permanent baby gates. It's to know which gates we opened, why we opened them and whether they should still be open.
The permissions nobody remembers granting
This part feels considerably less futuristic to me. I do it now with computers. An app needs access to a folder, microphone, network or some other part of the system, so I turn the permission on. Six months later it's still there and I have only the faintest recollection of why.
Software accumulates access. Organizations accumulate exceptions. Governments accumulate rules. Safety procedures accumulate because removing one feels considerably more dangerous than leaving it alone. Eventually nobody asks whether the original reason still exists.
Safety isn't something we install once. An immune system certainly doesn't work that way. It responds to current conditions. It can become weaker or overreact. Something harmless can become dangerous; a threat that once required an enormous response may no longer require one.
Protective systems need reality checks too: does this still make sense under current conditions? Does the software still need that permission? Does the restriction still solve the problem that justified it? Did we create an exception during an emergency and quietly turn it into normal policy?
Government is particularly bad at this because rules have a tendency to become objects once created. There is an understandable asymmetry. If removing a rule is followed by something terrible, somebody owns that decision. Leaving an old rule in place often requires no new decision at all.
That is how protection can acquire authority by inertia. Nobody necessarily chose today's arrangement. People chose yesterday's safeguard and forgot to revisit it.
The autoimmune problem
This is where I like the immune-system comparison better than the usual safety metaphors. Nobody has to be persuaded that immune systems matter; they keep us alive. Nobody has to be persuaded that more immune response is therefore always better either. An immune system that attacks too little is dangerous. So is one that attacks too much.
Protective technology can develop the same problem.
I recently moved much of my computer use from a Mac to Linux. Security wasn't the only reason, but I had become increasingly irritated by systems that were ostensibly protecting me while making ordinary things I had deliberately chosen to do more difficult.
Banking systems can produce the same reaction. Fraud protection is valuable right up until I am repeatedly prevented from making my own perfectly legal transaction because a system has decided my behavior looks unusual.
At some point babysitting becomes its own source of pain. Worse, people route around it. If the approved system becomes sufficiently burdensome, people find another system, disable protections, create workarounds or simply accept risks they wouldn't otherwise have needed to take. Now the safety mechanism has produced more danger.
That is the autoimmune problem.
The answer isn't no protection. That would be a peculiar lesson to draw from an immune disorder. It's proportionate protection: enough sensitivity to notice real danger without treating every unfamiliar thing as an attack.
Knowing what normal looks like
This returns me to those AI incidents.
AI companies plainly have enormous incentives not to have their models escape testing environments, access somebody else's systems or produce a spectacular incident followed by lawsuits, customer panic and regulators arriving with very large binders. But an incentive not to have a catastrophe isn't enough.
An organism has an excellent incentive not to die too. It still needs an immune system. Wanting to avoid disaster does not guarantee that you will recognize its first symptoms.
That requires knowing what normal looks like well enough to notice change. Which tools are agents actually using? Which permissions? Is behavior drifting? Did something unusual happen once, or is a pattern developing?
Increasingly, some of that observation may have to be done by other AI. Humans operate at human speed. Large numbers of autonomous agents don't necessarily have that limitation. So perhaps we get working agents, monitoring agents, adversarial testing agents, anomaly detectors and shutdown systems.
Fine.
Who watches them?
I don't think the answer can be one enormous Safety AI sitting above everything else with the authority to decide what all the lesser AIs may do. We have encountered that solution before in human institutions.
Give protective authority enough power and permanence and it tends to develop hobbies.
Better answers probably involve layers, independent checks, visible records and retained human authority. And provenance suddenly stops sounding like filing.
If three agents agree, are those three independent judgments? Or did they all inherit the same bad assumption? Did one influence the others? Are they actually separate agents or three identities produced by the same underlying system?
Three voices saying the same thing are not three pieces of evidence merely because they have different names.
We need to know who acted, what influenced the action, which tools were used and what changed. Otherwise even a safety system can manufacture the appearance of consensus.
Something will get through
There is another thing immune systems understand better than perfect rulebooks: sometimes something gets through.
I don't mean we should shrug at AI failures because perfection is difficult. I mean systems designed on the assumption that prevention will always work are terribly brittle when it doesn't.
Cybersecurity has never been solved once and declared complete. You detect, contain, isolate, restore, patch and learn. Then somebody invents something new and you begin again.
AI safety may look much more like that than like a final set of commandments. Preventing every failure may be impossible. Preventing one failure from taking everything else with it is a much more practical ambition.
That means backups, compartmentalization, rollback and redundancy. It means preserving enough evidence to understand what happened and actually testing recovery systems rather than admiring them on a diagram.
It also means accepting something I don't particularly enjoy accepting:
Safety is maintenance.
You don't finish it.
Somebody still has to decide
The more I thought about this, the less it seemed to be only an AI problem.
Doctors protect patients. Banks protect customers. Governments protect citizens. Operating systems protect users. AI increasingly protects or assists all of them. There is a recurring temptation to ask the protective system to make uncertainty go away.
Tell me which choice is safe. Warn me away from the dangerous one. Make sure nothing bad happens.
But a system cannot take responsibility for the outcome without acquiring enough authority to influence the choice. Those things travel together.
I'd rather have a doctor explain what is happening, give me the options and likely consequences, then say, “This is what I recommend, and here is why.” The recommendation matters. Expertise matters. So does the fact that it remains a recommendation.
I want something similar from AI systems and from the systems supervising them: show me the evidence, make the tradeoffs visible, warn me when I'm doing something stupid, and distinguish clearly between this is dangerous and you are forbidden to choose it.
Sometimes prohibition is justified. Some risks fall on other people. Some actions are too consequential to treat as purely individual choices. But that should itself require an argument, not merely the existence of a safety system capable of saying no.
There is no easy button that transfers responsibility without also transferring authority.
Healthy doesn't mean harmless
A healthy immune system doesn't promise that nothing will ever make us sick. It notices, responds, learns and recovers. Sometimes it gets things wrong. It requires maintenance, and its response has to remain proportionate to what it is trying to protect.
That seems increasingly like a useful way to think about AI safety. We need enough observation to notice trouble before the catastrophe announces itself. Permissions that reflect demonstrated reliability rather than raw capability. Independent checks, provenance and recovery. Periodic reassessment of safeguards that may have outlived their purpose.
And we need to remember that the safety machinery can become part of the problem.
The more responsibility we hand to systems meant to protect us, the more authority those systems will need in order to do the job. That isn't necessarily an argument against giving it to them.
It's an argument for noticing when we have.