Billing an agent run
In May 2026 I started Sukuna Headless, a managed agents API at Midsphere. A developer would start a run for one of their own end users, the run would execute in its own sandbox, and the developer’s workspace would pay for it from a prepaid credit wallet. The first commit, on 6 May, had a rules file with nine steps for what happens when an end user invokes the agent. Step 8 of business-rules.md is one line: “Subtract credits as the run progresses.”
The repository has 14 commits and stops on 25 June. This is how that one line turned into code, in the order the commits show it.
A wallet that only knew settled money
The first commit already had a wallet, an append-only ledger, and a check constraint keeping the balance at or above zero. It had no idea a run could be in progress. The wallet’s available() returned the cached balance, and its docstring said so: “settled, not authorised.”
The next two weeks of billing work went into the other side of the wallet, getting money in. Top-ups through Dodo Payments on 7 May, refunds and webhook ordering on 8 May, and on 19 May a rework that made Dodo the source of truth for credit grants and the local wallet a projection of it. The plan for that rework, still in todo.md, kept one property on purpose: the debit for a run stays local and atomic, and a background task tells Dodo afterwards.
None of that answered step 8.
The plan on paper
The answer was written down on 8 May, a month before any of it was code. That commit added nine documents under docs/coupling about how the backend and the sandbox would talk once they stopped sharing a database. The sandbox’s existing billing interface could create a hold, bill a tool call, settle and refund. The documents took all of that away from it:
The sandbox must not create holds, debit wallets, write ledgers, settle holds, or refund holds.
The backend would reserve credit before the run, the sandbox would enforce a limit locally and report signed usage, and the backend would settle. separate-db-open-decisions.md adds the sentence the rest of the work hangs on: “Billing must not depend on a terminal callback arriving successfully.”
separate-db-sandbox-contracts.md also lists what the backend would check a usage report against: the reserved hold, the lease maximum, the rate card, event sequence consistency, the known tool and model configuration, and the sandbox provider’s own record of runtime. todo.md records the contracts as frozen on 10 May.
What shipped on 6 June
The holds arrived in one commit on 6 June, with the rate card, the signed manifest, the callback endpoint and the first sweeper. available() changed to the balance minus the sum of active holds. Starting a run now computed a lease in orchestrator.py: a runtime limit from the end user’s profile or 3,600 seconds, 1,000,000 tokens, and a cost ceiling of the smaller of the available balance and $50. The same number became the hold.
Several things differ from the 8 May documents. The commits show the differences and not the reasons.
| Planned, 8 May | Shipped, 6 June | |
|---|---|---|
| Two starts racing for one wallet | a version column, optimistic locking |
a row lock on the wallet, then a recheck |
| Hold to run | foreign key | one to one |
| Rate card | per tool, model or storage key, with a markup | one active row per model tier, three rates |
| A failed run | settle the hold | release the hold |
| Checks on reported cost | hold, lease, rate card, sequence, runtime | the hold |
The last two rows are the ones that matter for money. The terminal branch in callbacks.py is short:
if isinstance(event, RunFailed):
hold.release()
else:
actual_cost = min(run.total_cost_usd, hold.amount_reserved_usd)
hold.settle(actual_cost)
The tests pin both outcomes with a $100.00 wallet. A completed run that reports $0.50 leaves $99.50. A failed run leaves $100.00. The lease contract reports a breached limit as a failure, so a run that hits its own cost ceiling is also free.
run.total_cost_usd is whatever the sandbox last reported. The rate card in rate.py has a compute_cost method and tests for it, and nothing outside those tests calls it. The clamp to the hold is the one check that shipped. The architecture document had already named that as the floor: a compromised sandbox “can report false usage” and the backend “caps at the reserved hold amount.”
The runs that never say they ended
The 6 June commit had one periodic task, reap_orphan_runs, which fails any run with no callback for its runtime limit plus 60 seconds. On 21 June I added seven more. Three of them close holds: one compares the provider’s list of live sandboxes with the backend’s, one cancels a run whose approval request sat unanswered for 300 seconds, and one sweeps holds past their expiry. A nightly task only counts mismatches and warns.
The reaper and those three end on the same rule, here from reconcile_holds.py:
if run.total_cost_usd > 0:
hold.settle(min(run.total_cost_usd, hold.amount_reserved_usd))
else:
hold.release()
So a run that dies silently is charged its last checkpoint, and a run that reports its own failure is charged nothing. The 8 May plan wanted the silent case settled from the last checkpoint plus runtime duration. Only the checkpoint is used.
What 25 June corrected
The last commits changed three things in the callback path, and the comments left behind say what was wrong.
The signature check failed against the sandbox’s signatures. The manifest carries the per-run key as a hex string, and the sandbox signs with the bytes of that string. The backend had been verifying with the raw key.
The ingestion loop treated any jump in sequence number as a lost event and stopped. The comment that replaced the check says sequence_no is “the sandbox’s GLOBAL runtime-event counter”, that only some events become callbacks, and that callback sequences are therefore “sparse (e.g. 1, 7, 18), not contiguous”. Under the old rule a batch numbered 1, 7, 18 would have been acknowledged through 1 and never further, so the terminal event would not have been applied and the hold would have waited for a sweeper.
A failure with an error code outside the backend’s list raised a validation error. The new docstring says that would “500 the callback and orphan the run”. Unknown codes now collapse to sandbox_internal.
The same commit set the sandbox’s own shutdown timeout to the lease’s runtime limit.
Where it stopped
| Date | Change |
|---|---|
| 6 May | Wallet, ledger, lease schema. available() is the settled balance. |
| 8 May | Design documents: backend holds, sandbox reports. |
| 19 May | Wallet becomes a local projection of Dodo’s credit balance. |
| 6 June | Holds, rate card, signed callbacks, settlement, first sweeper. |
| 21 June | Seven more periodic tasks, three of which close holds. |
| 25 June | Signature key form, sparse sequences, unknown error codes. |
Two pieces did not follow the 25 June change. detect_gaps.py still flags a run when it holds fewer events than its highest sequence number, which is true of any sparse stream. And test_gap_stops_processing still asserts the contiguous rule.
What I would do differently
Sign and number one real batch from the sandbox before writing the ingestion rules. The contracts were frozen on 10 May, and both the key form and the sequence numbering were wrong until 25 June.
Recompute cost from tokens with the rate card at settlement, as the 8 May document said, and compare it with what the sandbox reported. The table and the method are there.
Decide once what a run that stops early owes. A reported failure is free and a silent death pays its last checkpoint, and the record does not show that I chose that difference.