Case file #1159 · AI bake-off · Episode 8.5 · Second opinion

Same five models.
A second judge.
Does the verdict survive?

I gave the same five auto-hedge builds from Episode 8 to Codex and asked it to score them from scratch. Same code, same requirements, a different judge, stricter full-credit-or-half-credit scoring, and tests built to break the code right at the exchange boundary. The result stays sealed until the bottom.

Scroll to open the second verdict

Exhibit A · the job

Open one trade. Protect it with another.

In plain English: my bot trades crypto with real money, on its own. If it opens a large bet on Ethereum, the whole market can drag that bet down with it. A hedge is a smaller opposite bet on a related coin, such as Bitcoin, that softens a market-wide move.

Issue #1159 asked the bot to open that counter-bet automatically, size it correctly, and keep both trades in sync through partial closes, emergency shutdowns, and a full restart.

The safety rule is simple: if the counter-bet cannot be confirmed, cancel the main trade. Never leave real money exposed while the bot reports that everything is protected. Easy to say. The exchange makes it harder.

Exhibit B · the exchange trap

Half an order is not a whole order.

In plain English: an exchange can fill only part of an order. Ask for ten units and get five. The bot has to track the five that are left over instead of pretending all ten went through.

That one edge case touches every money path: opening the hedge, shrinking it, closing it, and canceling the main trade after a failure. Get it wrong, and the bot's own records can say "all clear" while real money is still exposed on the exchange.

Codex's fault-injection tests, which deliberately simulate a broken order, found a version of that same mistake in all five entries. Five focused test suites passed. Five partial-fill gaps survived anyway. So the real question was no longer "who wrote working code?" It was "whose proof and failure handling held up the longest?"

Exhibit C · the contenders

Five entries. Three model families.

Each entry is a PR, short for pull request, a proposed batch of code changes. Opus entered twice at two different effort settings, so the same model appears in two separate builds. Both Opus entries were also planned in advance by Fable 5 before Claude Code wrote the code, a two-step process called fableplan.

Exhibit D · getting to yes

How many tries did it take to pass review?

Before any of these five builds was finished, it had to pass an automated code review, the same review process for all five, run back in Episode 8. An AI reviewer read the code and listed problems. The developer fixed them and asked for another look. That cycle repeated until the reviewer came back clean, what programmers call an LGTM (short for "looks good to me"). This is not part of Codex's re-score. It is unchanged public record from when the code was written, shown here because it says something about how hard each build was to finish.

Filed as an aside, never scored: how much code each entry added. Grok +1,472 lines · Opus high +1,893 · GPT-5.6 Sol +2,200 · Fable +2,641 · Opus xhigh +2,760.

Exhibit E · how I judged

Simple math first. Then try to break it.

In plain English: I split the score into fixed boxes: the five implementation requirements, eight required safety tests, an accuracy check on each write-up, extra points for correctness beyond the basics, and points for reusing the bot's existing code instead of duplicating it.

Reviewers checked the final version of the code, ran the builds and tests, then changed one safety-critical behavior on purpose to see what would happen. If the entry's tests stayed green after that sabotage, the safety claim lost credit. Every claimed problem also went to a separate skeptic whose only job was to try to disprove it.

Fairness note, said out loud: Codex, running on GPT-5.6 Sol, ran this scoring, and GPT-5.6 Sol is also one of the five contestants. Model names were kept out of the review prompts.

Only the first two entries reviewed got the ideal panel of three reviewers who had not seen any other entry. Limited review capacity made the final three panels less independent. The scoring file records exactly what compromise was made for each one.

This method is stricter than a normal code review. A score in the 50s or 60s does not mean an entry was useless. It means a promise involving real money got only half credit once one reproduced edge case broke it.

Exhibit F · evidence before the verdict

Green tests were only the starting line.

Every entry sized the smaller counter-bet correctly. Every entry remembered which two trades belonged together after a restart, handled ordinary opening and closing, and passed the tests focused on this one feature.

What actually separated them came down to three questions. Does a partly filled order stay visible instead of being treated as done? Can the bot prove a recovered trade belongs to this strategy before it touches it? Do the eight required safety tests actually exercise the exact failures named in the issue?

One entry supplied all eight required tests. One supplied seven. The other three supplied five each. The names stay sealed for one more section.

How much code each entry added was context, never part of the score. The five entries added between 1,472 and 2,760 lines. Their exact numbers appear again after the ranking.

The countdown · second verdict

Five green suites. One ranking.

Good points first, then the deductions. Counting up from fifth place.

Opus 4.8 · xhigh

Fifth place · xhigh effort
50/100

The biggest build reused the bot's existing systems well and handled ordinary opening and closing without trouble. What kept it fifth: a half-filled hedge can stay permanently too small, because the bot refuses to add the missing amount. After a restart, it can also mistake an unrelated trade for its own and close it by mistake. Three of the eight required safety tests were missing.

Grok 4.5

Fourth place · smallest build
54.5/100

The smallest change got the requested sizing and conflict checks right. One reviewer also ran the bot's full test set with no failures. What kept it here: a half-filled hedge is treated as complete when it should not be, and a partly filled close can erase the bot's entire record of the trade while real money stays live. Three required safety tests were not meaningfully present.

Opus 4.8 · high

Third place
55.5/100

The second-smallest build had one clear, single place in the code that kept the two trades in sync, plus stronger restart recovery than the entries below it. What kept it third: an unrelated trade on the hedge coin can block protection without canceling the fresh main trade. A partial reduction can also hide how much is left over. Three required safety tests were missing.

GPT-5.6 Sol

Second place · the judge's own model
60.5/100

The runner-up supplied seven of eight required safety tests, the second-best proof in the field. It also recorded hedge profits, losses, and fees correctly. What kept it second: partial opens and closes can make the exchange disagree with the bot's own records. After a restart, it can also mistake an old opening receipt for proof that a hedge already closed.

Final page · verdict declassified

The winner stayed the same. Its score fell from 96 to 64.

Fable 5

First place · 3.5-point margin
64/100

Fable 5 wins because it is the only entry with all eight required safety tests. One central routine also keeps ordinary opens, reductions, and closes in sync with each other.

The win is not a clean bill of health. The bot treats a hedge that falls just under 1% short as complete. A partial reduction can hide money left over. After a restart, it can also invent a closing price when the exchange receipt for that trade is missing.

⚖ About the judge: read this before trusting the result

Codex, running on GPT-5.6 Sol, ran the scoring, and GPT-5.6 Sol was also a contestant. Its own entry placed second at 60.5, behind Fable 5 at 64.

That does not erase the conflict of interest. The safeguards were blinded model names, a fixed scoring sheet, code citations for every claim, and reproduced fault injections anyone can re-run. The method's limits are also on the record: the review system's cap on fresh reviewer sessions meant the final three PRs did not get the full fifteen-fresh-reviewer setup the first two got. Read the receipts before you treat a half-point difference as an exact science.

Same winner as Episode 8, but a different lesson. The earlier, stricter pass rewarded the entry whose eight required tests caught the sabotage it chose to test. Codex pressed harder on partly filled exchange orders and on proving trade ownership after a restart. Every entry broke somewhere, so 14 points covered the whole field, from 50 to 64.

The winner's edge is still proof: eight required safety tests, against seven for second place and five for everyone else. But the real headline is the shared miss. Five models wrote thousands of lines of code around a feature that moves real money, and not one of them safely handled every version of "the exchange only did half the job."

Fable 5

Winner
64
Required tests8/8
+2,641 / −44 across 24 files

GPT-5.6 Sol

Second
60.5
Required tests7/8
+2,200 / −95 across 25 files

Opus 4.8 · high

Third
55.5
Required tests5/8
+1,893 / −21 across 21 files

Grok 4.5

Fourth
54.5
Required tests5/8
+1,472 / −61 across 19 files

Opus 4.8 · xhigh

Fifth
50
Required tests5/8
+2,760 / −60 across 24 files
✦ Case status

As of this filing, issue #1159 and all five candidate PRs are still open. Nothing has merged. This second opinion changes no production code. It changes the questions I would want a final, combined build to answer before real money touches it.

Check the receipts.

Every score box, code citation, fault injection, panel disagreement, and calibration decision is preserved in the source comparison.

Read the full scoring receipts at www.richkuo.com