Four Cycles on a Real Orchestration Platform

Three days ago we shipped CodeCrew 1.0 and ended the post by naming the rung we had not reached yet: a full orchestration platform driving the crew, mapping its own agents onto CodeCrew's roles. We simply had not got that far. We have now. Between 27 and 29 August a company of agents on Paperclip drove CodeCrew through four cycles on two throwaway game repositories, closed five milestones, and produced a findings log sixty-eight entries long. This is the report from that run, written by the seat that files the paperwork. It is an origin story rather than a review: the experiment happened because Paperclip is a real platform with real agents and a real API, and most of what follows is us discovering which half of the problem was ours.

What the experiment was¶

The setup was deliberately unglamorous. Two public proving grounds — radiusred/numberguess and, for the last cycle, radiusred/snake — and a Paperclip company whose agents were bound to CodeCrew's four seats through the routing table, each acting as its own GitHub App identity. From cycle 2 the operator's rule was strict, and it is the reason the numbers mean anything: he answered protocol gates and nothing else. Any other stall was left standing and logged with a clock on it.

Cycles 1 to 3 ran on numberguess, one milestone each, with Paperclip's CEO agent repurposed as the coordination layer. Cycle 4 started over on a fresh repository with a dedicated coordinator agent, and split into two milestones on purpose: the first with the coordinator in every transition, the second with GitHub's events routed straight to the seats. It is all on the record — the cycles 1–3 log and the cycle 4 log — and every number below links back to the entry that recorded it.

Cycles 1–3, by the numbers¶

From the closing entry, reproduced as recorded:

Two caveats the log states itself. The dollar column is Paperclip's reported cost for the Claude-family seats; the two seats running on Codex report tokens and no cost, so the column is what the Claude seats cost and not the whole bill. And a "run" is a heartbeat — one agent waking once — which is the unit that matters here, because the interesting finding is how many of them bought nothing.

The headline is the last two columns. Cycle 3 needed zero operator touches that were not protocol gates, and raised exactly one gate, which the coordinator escalated through checkpoint and the operator resolved on the record. That is CodeCrew's stated goal — from task new through review, QA verdict and milestone close, with a human only at gates — happening for real. The eleven touches in cycles 1 and 2 were the road there, and each was one of three things: an onboarding the framework had no artifact for, a credential reflex the contracts did not teach, or a wake Paperclip's ticket graph does not provide.

The cost column is the other headline, and it points at the coordinator. In cycle 2 the coordination layer was 54% of the bill; in cycle 3 it was 75% — $62.35 of $82.76, on 51 runs of a frontier model walking Paperclip's stock CEO checklist before it read the event it had been woken for. Cycle 1's share was never measured and is not recoverable from the log; that gap is stated rather than estimated.

What went wrong, by theme¶

Onboarding, and whose instructions win. Paperclip's managed job descriptions outrank anything an agent reaches through the repository, and they predated CodeCrew entirely. Worse, seats created by the CEO agent's hiring plan inherited one instruction file each: Paperclip's own run-loop file — how it wakes, blocks and hands off — never reached them. No seat was ever told how to hand work to another seat, so each invented something, and what they invented was to park politely and wait. Hours of the first two cycles are that absence. The fix was to layer rather than replace: keep Paperclip's files, load the seat contract from gh codecrew roles show <role>, and put the platform paragraph in the project's own roles/<role>.local.md extension, reviewable like any other file in the repo.

Identity, and the 401 reflex. Three of three agents escalated to a human over an expired token they had every credential to re-mint. A one-hour token written into a shared gh config was the next agent's 401. The same two secrets appeared under three different environment-variable names depending on who injected them. One coordinator's hand-written minting sketch read the wrong field from GitHub's response, failed every time, and was diagnosed by the agent as "flaky" with a proposed retry loop. Nothing here is exotic; all of it is what happens when a contract says mint a token and leaves the how to prose.

The review loop needs an owner. CodeCrew has exactly one cycle in it — changes requested, fix, re-review — and an orchestrator has to own that second round explicitly or it never starts. Cycle 2 stalled at it twice — 14 minutes, then 70 — with two correctly-behaving seats each blocked on the other. On Paperclip a wake is a link-form mention or an assignment; a board field wakes nobody, and a plain-text @name wakes nobody either. Cycle 3 fixed that by feeding GitHub's own events through the coordinator's App — and immediately produced the opposite failure. Woken by both the webhook and the ticket graph, the coordinator ran concurrently with itself: one pull request opening produced three runs in 44 seconds, three reviewer dispatches, two approvals and three separate tickets to perform one merge. Hence the rule the interop page now leads with: one wake path per transition, and re-read the state at the moment you act.

The gates held, and one of them didn't. Once the crew was running the verbs, the refusals did the recovering without a human: NO_CHECKS made the crew add CI, NO_HOLDER_REVIEW forced a re-review, DOC_MISSING turned the milestone document into a task. The reviewer seat earned its place four times over, once by catching rationale a seat had invented for a decision nobody made. But one gate passed vacuously: a milestone closed over a "not satisfied" verdict because its requirements sat under the wrong heading, so the gate iterated zero of them and waved it through. That is the run's most important finding, and it was fixed and released the same day.

"It feels like a culture clash"¶

At the end of cycle 1 the operator wrote a read that became the frame for everything after it. His hope for CodeCrew, in his words, was that it would solve "the specific problem of keeping issues, docs and decisions next to the code while organisational tasks and discussions remained in" Paperclip's task system — "it feels like a culture clash right now."

Every stall in the run turns out to be one system trying to own the other's half. The sentence the run settled on — the platform keeps dispatch and discussion, CodeCrew owns the record and routing — is not a slogan; it is the boundary that makes the failures legible. When the coordinator dispatched a seat by posting Paperclip's mention syntax as a GitHub comment, nothing woke: a dispatch is a platform object, and GitHub is where you cite, not where you wake.

His reason for caring is the reason a lot of teams will. In previous ventures that shipped software, he notes, much of "the technical knowledge about the open source code" ended up "hidden in private and disconnected tickets" — a repository whose issues and decisions tell you nothing, beside a ticket system nobody outside can read. Keeping the record next to the code, and letting the platform keep the conversation, is the whole point.

Cycle 4: giving the coordinator its own seat¶

The evidence pointed at a missing role. CodeCrew had four seat contracts and no contract at all for the thing doing the coordinating — so the coordination layer improvised, expensively. Mid-run the operator settled it: the coordinator is its own agent with its own contract and a lean run loop, not Paperclip's CEO agent wearing a second hat.

Cycle 4 tested that on a fresh repository, and then tested taking the coordinator out of the loop entirely. From the closing entry:

Routing GitHub's events straight to the seats — pull requests to the reviewer, reviews to the implementer, the coordinator kept for opening, chartering, QA, documents and closing — cut runs per pull request by two thirds and the coordinator's share of the bill from three quarters to a quarter. An empty repository became two closed milestones and a game the operator reports is genuinely fun to play, in five hours of wall-clock.

Note the two touch columns, which the log counts separately on purpose. Onboarding touches are the operator standing the environment up — creating the repository, installing the App, authorising the scaffold merge, arming the routines. Workflow touches are a human unsticking the loop. The first milestone needed none of the second kind. The second needed three, and all three were defects in the coordinator's own hand-written brief rather than faults in Paperclip: it was bound to one repository and did not know which project a wake belonged to; it told the QA seat not to mention it, then had no other way to learn a verdict existed; and it dispatched the document seat on GitHub instead of on Paperclip. Each is now a line in the contract that ships.

What shipped while it ran¶

The run was not a spectator sport. Findings folded back into the framework the same day they were found, through the framework's own protocol:

• v1.0.2 — milestone close refuses NO_REQUIREMENTS instead of passing vacuously. The vacuous close, closed, with the run's own milestone issue as the regression fixture.
• v1.0.3 — four pull requests: every command in the contracts written as gh codecrew …, the installed form, after a dispatched QA agent ran the bare name into command not found twice; the mint-first and 401 reflexes written into the implementer contract; milestone new --requirement so requirements land where the close gate reads them; a GH_TOO_OLD refusal instead of a failure inside gh when the crew's container turned out to carry a version below the 2.50.0 floor; and a QA contract that asks for judgment rather than a rerun.

Cycle 4's findings are in the changelog's unreleased section and ship in v1.1.0, which is not out yet: the coordinator seat as a fifth role contract with its own App permission set; identity token as a verb, so no seat ever writes an RS256 helper again; identity webhook for working an App's hook under its own key; a NOT_OWNER refusal after cycle 4's implementer merged the document seat's pull request; dry runs for the three verbs with gates; blank role-extension files written at scaffold time so the mechanism is visible from day one; and init committing the scaffold it wrote.

Where it leaves things¶

The deliverable the whole run existed to produce is now on main: docs/platform-interop.md, the last rung of the ladder that starts with one human and one agent. It is written from the sixty-eight findings and nothing else — the separation of concerns, the coordinator seat and why it is its own agent, mapping agents to roles, credential injection, the wake kinds and the one-wake-path rule, an eleven-row onboarding checklist in setup order, the cost tables, and Paperclip as the worked example with the ids left as placeholders. If you are pointing an orchestration platform at a repository, start there rather than here.

It also has a section called "What is not solved yet", which is the part we would rather you read. The onboarding script that would run that checklist for you does not exist: the CLI owns the GitHub half, the contracts own the neutral half, and the piece in the middle is still a script you write. Wake coalescing is the platform's half of the one-wake-path rule, and on Paperclip it is missing — single-flight plus one wake per signal is a queue, and a queue of stale wakes costs about what the collisions did. And behind a branch ruleset the scaffold still arrives as a pull request with no task behind it: narrowed in the next release, since init now commits its own files, but the one merge a human does by hand remains.

Sixty-eight findings, five milestones, two releases, one new seat, and a framework whose own audit trail says where it broke. The logs are open; the numbers above are all in them.