BunBunLabs · Notes

Pay for verified work, not tokens

Tue, 07 Jul 2026 00:00:00 GMT

A token is a strange thing to sell. It is the input, not the result, and for code the two come apart hard. A model can spend a great many tokens producing a change that does not compile, and very few producing one that ships. Priced by the token, the customer pays the same for both, and pays the most for the runs that wasted the most. The incentive points the wrong way.

The honest unit is the outcome: a change that is verified and accepted. That is what a person actually wanted, and it is the only thing worth charging for. But pricing purely on the accepted result has its own trap, because verified work carries a real cost that someone paid, whether or not the result was the one the customer hoped for. Real compute ran. Tests ran. Critics ran. Charging nothing for honest, working effort is neither sustainable nor, in its own way, honest.

So the charge is designed to follow the outcome in two plain layers. If real, checkable work happened, if the change compiled and the tests ran, you pay the cost of that compute, whether or not you keep the result, because the work was real. On top of that, and only if the change passed verification and you accepted it, sits a margin. And there is one ending that costs nothing at all: if the attempt crashed and produced nothing verifiable, the cost is ours, not yours.

What is meant to make that offer possible is the architecture, not a generous mood. A change is authored by one model, then attacked by critics from families that fail in different shapes, and checked against a verifier that is not a model at all but the compiler and the test suite. The point of that machinery is to make one ending, a crash that produced nothing, rare, so honest work should land in the verified, paid range rather than the unpaid one.

There is a longer game underneath. Every one of those decisions, the change, the outcome, the correction, is kept. The point of collecting it is to train models that reach the verified range more often on their own, that find a working answer sooner and waste fewer attempts getting there. The pricing and the data are the same loop seen from two sides: pay for what works, and learn from the difference between what worked and what did not.

We do not publish which models do the authoring or the checking, or the numbers behind the floor and the margin. The rule is simple: you should pay for verified work, never for a crash, and a premium only on a result you chose to keep.

Standing on the frontier

Tue, 07 Jul 2026 00:00:00 GMT

It should be said plainly, because the opposite is easy to assume of a lab building its own models: we are not trying to beat the frontier. The largest general models are extraordinary, and we use them every day. They are the material this is built from and, in a real sense, the teachers. The agents that run our operations, and generate the data we intend to learn from, run on them today.

What we are building is a different thing, and it sits downstream. A frontier model is a generalist of remarkable range, and range is expensive. Set to a narrow, repetitive, cost-sensitive job, it is often far more capability than the job needs, and paying for that surplus, in money and in energy, adds up across millions of small tasks. The sensible answer is not a bigger generalist. It is the right tool for the job: keep the powerful general model for the genuinely hard problems it is uniquely good at, and run a small, specialised model, trained for one vertical, on the rest.

None of that is a quarrel with the frontier. Building a specialist for a domain is only possible because a generalist got good enough to run the business that produces the domain’s data in the first place. The frontier does the hard, open-ended reasoning. The specialist does the repeated, well-defined work underneath it, cheaply and reliably, and the hard cases escalate back up.

There is also a practical reason, not only a principled one. A company that positions itself against its own suppliers is telling on itself, and a lab that needs the frontier to exist has no business pretending it is the enemy.

The specialised models this is aiming at do not exist yet. What exists is the substrate that would train them, and the conviction that the future of applied AI is not only larger generalists but also smaller, sharper, cheaper specialists, standing on the frontier the large models built.

The unit of account

Sun, 05 Jul 2026 00:00:00 GMT

A token is priced in fractions of a cent, and that price is the one everyone quotes. It is also the wrong unit. What a system actually costs is the cost of an answer you can accept, and a single unverified answer that turns out wrong does not save you that cost. It defers it. The retry, the review, and the debugging that follow a wrong answer shipped are the same expense, billed later and under a different name.

Software has priced this for a long time. Boehm and Basili’s 2001 defect-reduction list put avoidable rework at roughly 40 to 50 percent of the effort on a project, and noted that most of it traces to a small fraction of the defects. The lesson is not that rework is unavoidable. It is that the cost of a wrong answer lands downstream of the moment you produced it, so a ledger that stops at generation is not measuring the work.

AI-assisted development has made this visible at scale. Faros AI’s 2025 telemetry across more than ten thousand developers found that heavy AI users merged 98 percent more pull requests and completed 21 percent more tasks, while review time rose 91 percent and the metrics that measure whether software actually ships stayed flat. The output moved; the throughput did not. The 2024 DORA report put a number on the same effect: for every 25 percent increase in AI adoption, it estimated a 1.5 percent decrease in delivery throughput and a 7.2 percent decrease in delivery stability. More was generated and less was delivered. The volume did not disappear. It moved into review, where a person now reads it.

The answer is to verify before you ship, and verification is not free. It is a line item, and it has to earn its place. The cascade literature makes this explicit. FrugalGPT sends a query to a cheap model first and escalates only when a scoring gate rejects the cheap answer, and RouteLLM reports over 85 percent cost reduction on one benchmark at 95 percent of the strong model’s quality by routing only about 14 percent of queries to it. The gate is the mechanism, and the gate costs something of its own. It also has to be good. The JETTS benchmark found that model-based judges are competitive when reranking finished answers but worse than step-level verifiers at guiding a search, and that their written critiques did little to make the generator better. A judge you cannot trust is worse than no judge, because it accepts confident wrong answers with a straight face.

Here is the case against us, and it is real. Verification does not always pay. A controlled study of code execution in program repair let strong agents run the code they wrote, which is the strongest gate there is, and found it moved the resolve rate by about a point, 63 percent against 64 percent, and in some configurations the agents did better with execution turned off. The gate cost more tokens and more time and bought almost nothing. Below a certain stakes this is the normal result: the verification layer costs more than the failures it prevents, and the honest move is to leave it out. We will not pretend otherwise. Verification pays when the outcome is expensive to get wrong and cheap to check against something real. That is a narrow condition. It is the one we build inside.

So we account differently. The unit is not the token. It is the accepted outcome, and an outcome is accepted when it passes a check against executable ground truth, not when a model is confident about it. That definition is the whole discipline: it puts the cost of being wrong back on the same line as the cost of the attempt.

How that nets out on our own traffic is a measured claim, and we do not publish measured claims about ourselves. ◷ It goes public when a client can attribute its own number. Until then this is an argument, not a result, and we would rather you read it as one.

When inference has to pay for itself

Sun, 05 Jul 2026 00:00:00 GMT

There is a running argument about whether the money going into AI will come back. Sequoia framed it as a $600 billion question in 2024: the revenue the industry would need to justify what it is spending on infrastructure. Goldman Sachs asked it more bluntly the same year, setting roughly a trillion dollars of coming capital spending against the benefit it has produced so far. We do not have a view on the macro number, and a two-person lab asserting one would be noise. The question that reaches us is smaller and harder to dodge: when the spending has to be justified deployment by deployment, which deployments survive.

It helps to be precise about where the losses actually are, because “inference is subsidized” is mostly false where people say it. Metered serving of tokens appears to be profitable. When DeepSeek disclosed the economics of its own inference system in early 2025, it reported a theoretical daily cost near $87,000 against a theoretical daily revenue near $562,000 at list prices. It was careful to call the margin theoretical, because its real revenue was lower: free web and app access, off-peak discounts, and no research or training cost in the figure. The point survives the caveats. Serving a token at list price clears the cost of serving it. The subsidy lives elsewhere, in flat-rate consumer plans where the heaviest users cost more than they pay, and in the research budgets of the frontier labs. So the reckoning is not a coming spike in token prices. It is each piece of deployed work having to show it was worth running.

On that, the record so far is poor. MIT’s NANDA initiative reported in 2025 that after thirty to forty billion dollars of enterprise spending, about 95 percent of generative-AI pilots showed no measurable impact on the bottom line. The study is survey and interview work and has been criticized on that ground, and we cite it with the caveat, but the shape is not in dispute: most deployments cannot show their return. The reason is usually not the model. It is that no one established what an accepted outcome was, or what it was worth, which is the subject of a separate note. A deployment that never defined its unit of account cannot prove it paid for itself, whatever the model costs.

The strongest argument against building for efficiency is that efficiency is about to stop mattering. Token prices are falling fast. a16z described roughly a tenfold drop per year in the cost of a given level of capability, and Epoch AI measured the same trend at anywhere from 9 to 900 times per year depending on the task, with a median near 50. If the price of the work halves and halves again, the savings from doing it efficiently shrink toward nothing, and raw token efficiency becomes a commodity rather than an advantage. We concede this plainly: as a way to spend less on tokens, our architecture has a shrinking edge. What deflation does not touch is the cost of a wrong answer accepted, which is paid in the workflow and not in the token, and does not fall when tokens do. There is a second concession worth stating in the same breath: cheaper and more reliable inference generates its own demand, so the capacity being built may well find work. We are not betting against AI. We are betting on which part of it pays.

So our assumptions are stated, and they are checkable against us later. The system is designed to be worth running at list-price inference, with no subsidized tokens holding up the math. The claim it rests on is that verified work inside a chosen vertical costs less per accepted outcome than an unverified call, at any token price. If token prices fall to nothing and that per-outcome cost does not follow them down, this note was wrong, and the errata will say so.

Collecting the right data now

Mon, 29 Jun 2026 00:00:00 GMT

BunBunLabs is going to train its own vertical models. It has not yet. No vertical models exist yet, and training them is gated on compute and on the data infrastructure that turns a live record into a training corpus, which we do not have today. What we do have, and what is easy to start recording too late, is the part that has to come first: the data itself, captured now.

Every governed decision the fleet makes leaves a trace: what was attempted, what the world did in response, and how it was corrected. An action, an outcome, a correction. Recorded together and verified, that triple is the unit a model can actually learn from: not only the chosen answer, but the rejected one beside it and the reason the difference mattered.

The discipline is to capture it at full resolution from the first day, because resolution is the one thing you cannot add back later. If today you record only that a check “passed,” you can never recover how close it came, what exactly it caught, or what the rejected version would have done. The signal exists at the moment of the decision; if it is not written down then, it is gone, and a corpus captured too coarsely cannot be retrofitted. You would have to live the traffic again.

So the honest state is this: the pipeline that records the work is live and generating data now. Across the agents already wired, a governed decision is recorded the moment it is made, at full resolution, and the machinery that binds an action to its real outcome and its correction is built and running. The full reasoning behind those decisions flows through observability we run ourselves, and we have proven a method for linking a decision to the trace that produced it, and are now wiring it in across the agents. What has not happened yet is the loop closing: today we collect traces, not yet complete training triples, because a complete triple needs a real outcome, and the first one lands when a real deal closes. Turning what the pipeline captures into a corpus, and the corpus into models, is the trajectory, and it is gated on compute and data infrastructure.

The schema, the storage, and the measurements that ride on top of it stay private. The principle does not: collect the right thing now, at the right resolution, so that when the compute arrives there is something real to train on, built from governed first-party decisions, not scraped and not synthetic.

Why two models fail differently

Mon, 29 Jun 2026 00:00:00 GMT

A frontier model is capable and bounded at the same time. Bounded because every architecture makes one trade in order to run at all, in how it handles the context you give it. Some compress that context to fit more in cheaply, and lose fine detail. Some select a subset to read at full fidelity, and miss what falls outside the selection. Some carry a running summary forward, and let the distant past fade. Some spread their attention evenly, and dilute it over a long input.

Each trade is reasonable. Each one also fixes, in advance, the kind of mistake the model will tend to make. The failure is not random. It is a property of the design.

The usual response is to ask several models and take the majority answer. It helps less than it looks. Models trained on similar data toward similar goals tend to be wrong in similar places, so their errors line up instead of cancelling, and a vote of correlated voices mostly multiplies the shared mistake. Past a handful of models, agreement stops buying correctness.

The opening is the opposite of agreement. If a model’s failure is a property of its design, then a second model built on a different trade fails somewhere else, and the first one’s blind spot is the second one’s clear view. The disagreement is the signal, and it is mechanical rather than lucky: a detail lost to compression is still there for full-fidelity selection; a fact that faded from a running summary is still present where it was selected exactly.

So the work is not to average models. It is to pair them across their failure mechanisms on purpose, to put each answer in front of the one perspective placed to catch what its author would miss, and then to settle any disagreement against something that cannot be voted with: a result you can run.

We do not publish which model plays which role, or how the pairings are chosen and tuned. That is the part that took the work. The idea is the easy thing to say and the hard thing to build, so we publish the idea and keep the build.