The system that won't be looked at

On 9 October the European Commission convened a special meeting of the scientific panel that advises the AI Office. The reason is stated plainly in the official note: recent incidents of loss of control, and technical developments on model monitorability and recursive self-improvement.

I read the recommendations presented by the panel's chair and vice-chair. One stopped me more than the others: stronger scrutiny of training techniques and architectures that reduce the monitorability of models.

In plain words: there are ways of building a system that make it harder to see inside. And the issue has made it into an official recommendation.

The law assumes it can understand

Every rule takes for granted something it never writes down: that whoever applies it can understand what it governs. The inspector walks into the factory and sees the machines. The judge reads the contract. The regulator examines the processing.

When the object stops being observable, the rule stays on paper but loses its grip. It doesn't become wrong. It becomes blind.

It is the same problem I have worked on for years, seen from the other side. In Habeas Mentem I write about the presumption of understanding: the law pretends that people understand what they sign, accept, click. They don't, and it can be measured. In Mente e Tecnologia, in an exploratory study, we used EEG and fNIRS to observe how a hundred people read legal documents. Complexity shows up in the data, not in opinions.

Here the fiction moves. It is no longer the citizen who "should" understand the contract. It is the authority that "should" understand the system. And this presumption is starting to crack too.

The burden lies with the designer

The answer I propose for people applies here as well. When someone designs an environment, the burden of making it understandable lies with the designer. Not with the person subjected to it, and not with whoever has to supervise it.

Applied to the most powerful models, this means something simple: monitorability is not a technical detail. It is a condition for being on the market. If a design choice reduces the ability to observe how the system behaves, it is up to whoever makes that choice to show the risk remains manageable.

Part of the framework already exists. The AI Act requires providers of general-purpose AI models with systemic risk to assess and mitigate possible systemic risks at Union level and to report serious incidents to the AI Office without undue delay (Art. 55(1)). According to the Commission, the AI Office has already requested and received serious incident reports and asked for information on security practices. Executive Vice-President Henna Virkkunen puts it this way: "The EU has the first law in the world that addresses systemic risk from AI".

Having the law is the first step. The second is being able to see what the law is meant to govern. An incident report is only as good as the ability to understand what happened. If the system is built to be opaque, the report arrives, but it explains nothing.

The panel also asks for governance of the risk posed by internal models, during development and when deployed for testing. That matters. Risk does not wait for the commercial launch.

Clarity, for those who supervise too

In legal design I often repeat one line: clarity, not simplicity. We don't ask a contract to be trivial. We ask it to be readable by whoever has to decide.

The same goes for systems. Nobody expects them to be simple. We can expect them to stay readable by those who must supervise them. We can expect the choices that make them unreadable to be disclosed, justified, verified.

After habeas corpus and habeas data, I argue we need a right to understand. This episode tells me that right has a second face. It is not only about people facing systems. It is about institutions facing what they are supposed to govern. A democracy that cannot look inside the tools reshaping it has already given something up, even if no rule has changed.

Understanding can be measured. Including the understanding of those who supervise.