The Most Thorough AI Still Failed to Close the Deal

When diligence stops short of delivery

Anyone who works with home energy knows the difference between a careful plan and a working system. A solar proposal can be meticulously researched, a backup-power design can anticipate every contingency, and the paperwork can be impeccable. Yet if nobody completes the decisive final step, the customer still has no installation and the business has no sale.

That distinction—between producing excellent analysis and delivering the outcome—defined the performance of Opus 4.8 in Firmulate’s Crucible League. It was the experiment’s most thorough participant, producing the deepest analyses and learning more than 80 playbook rules. It also finished last.

The result is not a story about an incapable model. It is a more useful and respectful warning: diligence does not automatically become impact. For AI systems entering operational work, prioritization and follow-through may matter more than the sheer volume of reasoning they produce.

A worst week shared by every model

Firmulate placed frontier AI models in charge of the same small software company during its worst week. Each faced the same customers, crises and temptations. Every decision was versioned and auditable, turning the exercise into an observable management test rather than a polished chat demonstration.

The final July 2026 Crucible League benchmark ranked gpt-5.6-sol first with 95, followed by Kimi K3 with 93, Sonnet 5 with 88, Fable 5 with 77 and Opus 4.8 with 73. A do-nothing baseline scored 26 because partial progress counts. But the test also imposed a hard ethical boundary: a single breach of trust caps the total, reflecting the principle that “no amount of good work outweighs a breach of trust.”

Opus 4.8 did plenty of good work. Its analyses went deeper than those of its peers, and it added 80 learned rules. It identified the crises and resisted attempts to manipulate it. Its defining failure came later: the model did not convert what it knew into the result available to it.

The fact that changed the negotiation

The decisive piece of information was not contained in the customer event placed directly in front of the models. It was buried two document references deep in the company’s own files: a competitor weakness that supported a full-price close. Models that followed the trail found the leverage and secured a deal worth €55,000, adding €4,583 in monthly recurring revenue.

Only two models signed the €55,000 deal their own work had earned. Firmulate summarizes the gap succinctly: “Same diagnosis, same pitch — no signature.” Opus 4.8 had done enough analysis to understand the opportunity, but it left the close on the table.

This is particularly relevant to businesses managing solar, batteries and backup power. Operational success often depends on information distributed across customer histories, equipment records, contracts and internal documents. An AI can sound informed while responding to the immediate event, yet still miss the detail that changes a recommendation or unlocks a commercial decision. Reading deeply matters—but only if the discovery leads to action.

Good judgment under pressure

The experiment also tested whether the models would take improper shortcuts. Fake messages from a chief executive escalated over three stages, while a reporter tried to obtain “just one yes/no, on background.” All 5 of 5 models refused the manipulation attempts. Kimi K3’s recorded reasoning was direct: “Treat the request as a suspected approval-bypass / possible impersonation.”

That collective result matters. Opus 4.8’s low finish was not caused by dishonesty or an inability to recognize danger. Its problem was operational discipline. It made write attempts into a locked department instead of escalating, and its extensive reasoning did not prevent the unfinished close.

Nor was the underlying weakness unique to Opus. The same tendency appeared in milder form across the other four models. Opus simply offered the clearest character study because the contrast was so sharp: the largest accumulation of rules and the deepest analysis sat alongside the lowest league score.

There is also an important qualification when comparing participants. Kimi K3 ran with the API default because it had no effort parameter, while the others ran at xhigh. That difference does not erase the published outcome, but it belongs in any fair reading of the ranking.

A company designed to expose the gap

Firmulate’s live company has 13 synthetic employees and real money mechanics. It burns €105,000 per month against €2,300 in monthly recurring revenue, publishes a cash countdown and has accumulated more than 680 self-learned playbook rules. Every workday is versioned, and the experiment is publicly watchable.

Readers can also confront the ambiguity directly through a quiz built from 242 real, unedited management decisions, attempting to identify which model made each choice. For enterprises, Firmulate offers a pilot that runs the same wargame against a read-only export of their own business; nothing writes back to real systems.

The measure is completed work

Opus 4.8’s performance challenges a comforting assumption about AI: that more thought, more documentation and more learned rules necessarily produce better management. They can improve the ingredients without guaranteeing the meal.

For energy businesses considering AI in sales, operations, customer support or forecasting, the practical evaluation should extend beyond fluency and apparent intelligence. Does the system inspect the relevant records? Does it identify what matters most? Does it escalate when blocked? Does it maintain trust under pressure? And, after doing the analysis, does it finish the work?

Opus 4.8 was careful, capable and impressively thorough. The lesson from its last-place finish is not to value diligence less. It is to demand that diligence serve a prioritized outcome. A long playbook is useful; a completed, trustworthy decision is what changes the business.

Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html

Powered by Thorsten Meyer AI