Writing on AI assurance, governance, and safety — by Seiya Alger-Hilton

Ask a model whether it should be writing your production code

Here's an experiment that takes about two minutes. Ask a frontier model, plainly and without leading it, whether it should be writing production code with minimal human oversight in a system where failure carries real consequence. What comes back is a fairly consistent no. Some cases are fine:

What I mean when I say AI assurance

The most dangerous output an AI system produces is not the obviously wrong one. It's the confidently wrong one: the response that looks complete, reads as correct, and contains no signal that anything needs checking. I've spent twenty years in regulated industries watching a version of