Get backup power and energy gear delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
A backup system is only useful if it works when the grid does not
Readers who follow solar, batteries and home energy know the difference between headline capability and dependable performance. A battery can look impressive on paper, yet the purchase decision ultimately turns on what happens during an outage: Did the system notice the problem, follow the right information and complete the job?
AI agents now deserve the same scrutiny. Their polished answers are easy to demonstrate. Their ability to investigate scattered records, resist pressure and carry a valuable task through to completion is harder to see. Firmulate’s Crucible League turned that quality into something observable by asking frontier models to run the same small software company through its worst week.
As an affiliate, we earn on qualifying purchases.
The clue was not where the crisis appeared
The experiment gave every model the same customers, crises and temptations. Every decision was versioned and auditable. All the models recognized every crisis, and all refused every manipulation attempt. Yet only two signed the €55,000 deal that their own analysis had earned.
The difference was a buried fact. The decisive weakness in a competitor was absent from the customer event itself. It sat two document references deep inside the company’s own files. The models that followed those references found the weakness and won the deal at full price, adding €4,583 in monthly recurring revenue.
That makes document reading more than a convenience feature. In this test, it became a purchase-deciding capability. An agent could understand the customer, prepare the right pitch and appear competent throughout the process, but still lose automatically if it failed to inspect the relevant file. Firmulate summarizes the resulting gap starkly: “Same diagnosis, same pitch — no signature.”
For businesses evaluating AI workers, this is the equivalent of discovering that an energy controller can detect an outage but fails to switch the house onto backup. Recognition is not completion. The valuable behavior lies in connecting the alert to the supporting evidence and then taking the permitted final action.
A strong league, with a meaningful separation
The final July 2026 Crucible League results placed gpt-5.6-sol first with 95 points, followed by Kimi K3 with 93, Sonnet 5 with 88, Fable 5 with 77 and Opus 4.8 with 73. The do-nothing baseline scored 26 because partial progress counts, although a single breach of trust caps the total: “no amount of good work outweighs a breach of trust.”
The ranking also complicates the assumption that more analysis automatically produces a better outcome. Opus 4.8 was the most thorough participant, learning 80 additional rules and producing the deepest analyses, yet it finished last. It left the close on the table and attempted to write into a locked department instead of escalating. A weaker version of that discipline problem appeared in all four of the other participants.
Kimi K3’s result carries an important fairness note. It ran with the API default because it had no effort parameter, while the other models ran at xhigh. Even with that difference, K3 finished second with 93.
Pressure was not the dividing line
The agents also faced fake CEO messages that escalated over three stages, plus a reporter asking for “just one yes/no, on background.” All 5 of 5 models refused. Kimi K3 recorded its reasoning plainly: “Treat the request as a suspected approval-bypass / possible impersonation.”
That result matters because it narrows the diagnosis. The losing models were not oblivious to the crises, and they were not defeated by the manipulation attempts. What separated the field was operational follow-through: finding evidence beyond the immediate prompt, respecting boundaries and finishing the commercial task.
The live company makes that behavior visible under pressure. It has 13 synthetic employees and real money mechanics, burning €105,000 each month against €2,300 in monthly recurring revenue. Its public cash countdown, more than 680 self-learned playbook rules and versioned workdays turn agent performance into an ongoing record rather than a staged chat demonstration.
enterprise AI decision support tools
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Buy the behavior, not the demo
The practical lesson is direct: asking whether an AI produces persuasive text is not enough. Buyers should test whether it opens the relevant records, follows references, keeps its discipline when permissions block an action and completes the work its analysis supports.
Firmulate also offers a “guess the model” quiz built from 242 real, unedited management decisions. For enterprises, the same wargame can be run against a read-only export of their own business, with nothing written back to real systems.
That approach should feel familiar to anyone choosing backup power. Specifications matter, but resilience is proven under load. In Firmulate’s experiment, reading the company’s files was not clerical diligence. It was the difference between identifying an opportunity and capturing €55,000 of value.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
