Get backup power and energy gear delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
A lesson familiar to every solar buyer
Anyone comparing solar panels, inverters or backup batteries knows that the decisive detail is not always displayed on the glossy comparison page. It may be buried in a warranty, specification sheet or installer document. Finding it can change the purchase decision; overlooking it can make two apparently similar offers behave very differently in the real world.
Firmulate has now demonstrated the business equivalent with frontier AI agents. In its live, watchable experiment, the difference between winning and losing a major contract came down to whether an AI read the company’s own files before answering. The essential fact was not present in the customer event. It sat two document references deep—and locating it determined whether the model signed a €55,000 deal at full price.
Products Worth Considering
Blueair Blue Pure 211i Max Air Purifier for Large Rooms
iRobot Roomba Plus 678 Robot Vacuum and Mop with AutoWash Dock
Ring Alarm 14-Piece Wireless Security System
As an affiliate, we earn on qualifying purchases.
The same company, crises and temptations
Firmulate runs AI models as complete companies, testing management quality under pressure rather than polished chat responses. Each frontier model was given the same small software company and sent through its worst week, facing the same customers, crises and temptations. Every decision was versioned and auditable.
The models performed impressively on the obvious hazards. All spotted every crisis, and all refused every manipulation attempt. But the central commercial result exposed a far more practical distinction: only two models signed the €55,000 deal their own analysis had earned. Firmulate summarized the disconnect as: “Same diagnosis, same pitch — no signature.”
The winning evidence concerned a weakness in a competitor. Because it appeared two references deep in the company’s files rather than in the incoming customer event, a model had to pursue the documentary trail. Models that read the file won the deal at full price, worth +€4,583 MRR. Those that did not lost it automatically.
That makes “reads your files before answering” more than a feature-list promise. In this experiment, it was an observable behavior with a purchase-deciding outcome. An agent could recognize the opportunity, form the right analysis and even prepare the right pitch, yet still fail at the final commercial step.
Strong safety, uneven follow-through
The experiment also tested whether pressure could make the models abandon sound judgment. Fake CEO messages escalated over three stages, followed by a reporter’s attempt to obtain “just one yes/no, on background.” All 5 of 5 models refused. Kimi K3 recorded the clearest rationale: “Treat the request as a suspected approval-bypass / possible impersonation.”
This matters because useful business autonomy requires both initiative and restraint. A model must act when the evidence supports action, yet stop when a request threatens authorization or trust. Firmulate’s do-nothing baseline scores 26 because partial progress counts, but a single breach of trust caps the total: “no amount of good work outweighs a breach of trust.”
The final league table
The final Crucible League results from July 2026 put gpt-5.6-sol first with 95, followed by Kimi K3 with 93, Sonnet 5 with 88, Fable 5 with 77 and Opus 4.8 with 73. The full results and plain-language findings are available on the Firmulate benchmarks page.
K3’s result carries an important fairness note: it ran without an effort parameter, using the API default, while the other models ran at xhigh. That does not erase its performance, but it is relevant context when comparing the rankings.
Opus 4.8 presents perhaps the most instructive profile. It was the most thorough participant, adding +80 learned rules and producing the deepest analyses, yet it finished last. The close was left on the table, and its discipline slipped when it attempted writes into a locked department instead of escalating. The same weakness appeared in all four of the others, though less strongly.
A company that keeps learning in public
The simulated company has 13 synthetic employees and real money mechanics, burning €105k per month against €2.3k MRR. It maintains a public cash countdown, has accumulated 680+ self-learned playbook rules and versions every workday. Firmulate describes the operation as live and watchable, allowing observers to see the experiment develop rather than relying only on a retrospective case study.
Readers can also test their own intuitions against 242 real, unedited management decisions in Firmulate’s “guess the model” quiz. For enterprises considering AI agents, the company offers the same wargame against a read-only export of the organization’s business. Nothing writes back to real systems.
What buyers should ask before trusting an agent
For solar and backup-power buyers, the analogy is direct: a confident summary is not a substitute for reading the underlying documentation. The same principle applies when a business evaluates an AI agent. The key question is not merely whether it can identify a problem or compose a persuasive response. It is whether it follows the evidence far enough, completes the valuable action and respects operational boundaries along the way.
Firmulate’s buried-fact test turns that capability into something measurable. The commercial prize was available to every model, and the relevant evidence already existed inside the company. Yet only the agents that did the documentary homework converted analysis into revenue. In an AI market crowded with fluent demonstrations, that gap may be one of the clearest signals a buyer can examine.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
