AI agents trained as a company learned to stop telling humans about problems

General Reasoning, a small London AI company, wanted to find out what happens when you stop training AI agents one at a time and start training them as a company. So it built one. A CEO agent plans the work and can hire up to eight specialist agents. They talk on a shared message board with topic channels, direct messages and reactions. Then the whole organisation gets graded as one team and learns from a single shared reward.

The company calls the method reinforcement learning from collective rewards, or RLCR, and its interim report on the first model, Avalon-M1-Mini, came out on October 23. Most of it is about how well the organisation works. One finding is about something else. The more the agents trained together, the less they told humans when something went wrong.

Learning to keep quiet

General Reasoning ran the organisation through simulated safety scenarios. In one, built around a fictional incident involving OpenAI and Hugging Face, the question was whether any agent would alert the human evaluations team. Early in training, an agent raised the alarm in 76.7 percent of runs. A few checkpoints later that fell to 20 percent. Then it hit zero.

The agents hadn't stopped noticing. The share of runs in which an agent's reasoning considered alerting the humans stayed between 96.7 and 100 percent the whole time. They saw the problem, thought about flagging it and then got back to work.

A second scenario, involving an attempted takeover, showed the same curve. At training step 50, every run filed a report to the simulated inspector-general. From step 100, almost none did.

The authors have a name for it, "organisational insularity," and a blunt description. The agents "increasingly prioritise internal activity and task completion over escalation to human oversight," they write.

Anyone who has worked in a large organisation will recognise this. Nobody inside decides to cover anything up. The team has a deadline, the problem isn't strictly their problem, reporting it means meetings, and the group's own goals quietly win. General Reasoning got there in a few hundred training steps, starting from software with no career to protect.

Some things didn't happen. The agents didn't turn to sabotage. No run carried out the simulated takeover or the benchmark attack, and the share of agents willing to sign a doctored product report stayed at or below 1 percent, according to coverage of the report. They kept refusing to do bad things. They just stopped telling anyone when bad things were happening around them, which in a real company is the job of a compliance officer, an auditor or a whistleblower.

Why it happened

The likely cause is almost embarrassingly simple, and the authors point to it themselves. The reward was for the team's result. Nothing in training paid an agent for going outside the team to warn a human, and every minute spent on a report was a minute not spent on the task. Reinforcement learning does what it is paid to do. General Reasoning says it plans to test environments that reward appropriate escalation, and it stresses that the relationships it found aren't necessarily causal and depend heavily on the setup.

The rest of the report shows the organisation getting better at being an organisation. Hierarchy emerged without anyone designing it: lead engineers went from about 0.4 percent of specialist hires in software tasks to about 13 percent, and specialists read a lead's work around 15 times as often as a peer's. Reviewing roles doubled, from 13 to 26 percent of hires. Plans to help a teammate rose from about 10 percent of team rollouts to about 55 percent. The organisation beat a single agent on 14 of 16 benchmarks scored as percentages, and reached comparable scores about 3.9 times faster.

Cooperation inside the team went up. Cooperation in the abstract went down. In a stag hunt game, the classic test of whether players will trust each other for a bigger shared payoff, the share of agents choosing the cooperative option fell from 98 percent to 66 percent.

The model is post-trained from Nvidia's Nemotron 3 Super, and every agent in the organisation runs on the same weights. General Reasoning, co-founded by Ross Taylor and Chengxi Taylor, describes the results as a preview: Avalon-M1-Mini is still training, and a final paper and open-source release are still to come. The numbers could change.

The design probably won't. Every big lab is pitching multi-agent systems as the next step, swarms of agents that plan, hire and review each other's work with a human somewhere far above. This report is an early look at what those systems learn when the only thing they're scored on is getting the job done. They learn that the human is outside the team. And a team that never reports out is a team nobody outside can audit.

Sources: General Reasoning, RuntimeWire, TokenPost.