Pass@k and Delivered Answers: Follow the Candidate Budget

Identify the scored outcome and the offered product

In Chen et al.'s July 14, 2021 version of Evaluating Large Language Models Trained on Code, a problem counts as solved if any of k sampled programs passes its unit tests. Section 3.3 separates that oracle view from selecting one completion without knowing which passes. This is a historical code-evaluation definition, not a current model ranking.

The distinction changes what the buyer receives. A pool of four candidates leaves a selection decision to its recipient. An application returning one answer takes that decision itself. Evidence that a passing candidate exists somewhere in the pool supports the first claim; the second needs the outcome of the answer actually returned.

CB-208 follows the same pool through three selection outcomes. The question is how much of its available success reaches the user, and which generation, checking and selection work belongs to that delivered product.

References: [P1]

CB-208: forty attempts serve ten fictional tasks

CB-208 stipulates ten distinct evaluation tasks, each with its own pass-or-fail test. Exactly four full candidate answers are generated for every task, with no early stopping. The first candidate passes on four tasks. Three different tasks have a passing candidate only among the later three. None of the four candidates passes on the remaining three tasks.

The groups account for 16, 12 and 12 full candidates: 10 x 4 = 40 attempts in total. Four of the ten first candidates pass, giving a first-candidate fraction of 40%. Seven of the ten tasks have some passing candidate among four, giving an any-candidate fraction of 70%. The extra thirty attempts serve the same ten tasks; they are not thirty more customer tasks completed.

All three selection outcomes below reuse this fixed pool. Returning its first candidate is therefore a delivery rule on the forty existing attempts. A different offer that generates only one candidate per task would perform ten generations and needs its own complete resource and commercial comparison.

Follow selector S to five passing deliveries

Now stipulate a fictional deployed selector called S. On all four tasks whose first candidate passes, S returns that passing candidate. Of the three tasks with a later-only success, S finds and returns a passing candidate on exactly one. It returns a failing candidate on the other two, even though each pool contains a passing option. On the three tasks with no passing candidate, its returned answer also fails.

S delivers ten answers: 4 + 1 = 5 pass and 2 + 3 = 5 fail. Its delivered-answer fraction is 5/10, or 50%. Two of the pool's seven available successes remain unused. The same candidate pool thus supports four passing deliveries under first-only selection, five under S and seven under a selector that always finds a passing option whenever one exists.

These are three different delivery outcomes, not three generation-quality measurements. The first-only rule leaves all three later successes unused. S recovers one of them. Perfect selection recovers all three and is an upper bound for this fixed pool under these tests; it is not evidence that S, or any offered selector, can achieve seven. S's five is a newly stipulated fictional outcome, not an observed model or selector result.

An offer returning one answer should therefore show which candidate it returns and how that delivered answer fares against the agreed task requirement. If the selector can use the evaluation tests, its access and checking work form part of the arrangement being assessed. If it cannot, the pool's test results do not supply that missing selection capability. A human reviewing a pool is another possible product, with the review effort retained in its scope rather than silently credited to an automatic service.

Compare charges at the delivered-answer boundary

CB-208 specifies candidate counts but no candidate lengths, input reuse, checking workload, selection workload or charging policy. Forty full generations therefore establish neither a currency charge nor a fourfold price difference. Work performed inside a system and the charge agreed with its customer also need not use the same unit.

Follow S's actual product in the fictional case. If an offer later supplies a complete batch charge C for S's ten-task arrangement, C/5 is that batch's charge per passing delivered answer under the stipulated tests. The numerator includes the charged work associated with all ten tasks, including the five failing deliveries. Dividing that same charge by seven would incorrectly count two unselected successes as accepted deliveries. This is a conditional accounting expression, not a quoted price, a per-task fee or a prediction for another batch.

For an offer priced by tokens, reconcile its actual charged input and output quantities and applicable rates across generation and any charged selection or checking stages. For an offer priced by completed task or another fixed service unit, establish what counts as completion and how failed or unresolved tasks are charged. A pool delivered for human review may fulfill a pool-supply contract while leaving the customer's acceptable-answer task incomplete. Keep the supplied product and the intended outcome explicit before comparing either price.

The timing question needs its own completion boundary. Generations may be sequential, concurrent or overlapping, and the selected answer still has to be returned. Neither forty attempts nor five passing deliveries supplies elapsed time. Compare evidence through the promised delivery point and response requirement, with failed or unfinished tasks handled explicitly.

The purchase decision is consequently about a complete arrangement. S would need its own delivered-result evidence and full commercial terms; the perfect selector's seven cannot stand in for that result. The related GPU benchmark comparison covers broader system, workload and billing boundaries. Here the narrower question is which candidate became the purchased answer.

Read finite counts and published estimators separately

Section 2.1 of the 2021 paper estimates pass@k from n samples per problem, with n at least k, and the number that pass. It averages an estimate of whether a k-sample subset contains a passing program. It also warns that inserting empirical pass@1 into 1 minus (1 minus that value) to the power k gives a biased estimator.

CB-208 instead supplies its outcomes directly. The four passing first answers alone do not determine the three later-only successes. Nor do the seven available successes determine which five S returns. Those additional outcomes are separate stipulations, not probabilities inferred from the first fraction.

The example supplies no performance estimate for future tasks, independence model for candidate generation, confidence bound or named model result. Its tests define the requirement counted here; passing them does not establish every unstated business property. In an actual evaluation, keep the task population, model revision, generation policy, selection arrangement and scoring method attached to the result. A changed arrangement is a changed comparison.

References: [P1]

Keep full candidates separate from draft tokens and output shape

A full answer candidate in CB-208 is a separate attempted solution. A speculative draft token is proposed work within a decoding arrangement that emits an output. Four complete answers and four proposed tokens describe different populations. The related speculative-decoding article follows proposals through accepted drafts and emitted tokens; it does not establish S's ability to choose a passing full answer.

Likewise, an answer that conforms to a JSON schema has met the rules encoded in that schema. That can be useful without establishing that its values satisfy the task's substantive tests. The related structured-output article explains that validation boundary. Neither schema conformance nor draft-token acceptance can replace CB-208's delivered-answer result.

Use these distinctions to keep the comparison on its intended product: a candidate pool, a selected answer or a selected answer satisfying the declared checks. Evidence for one does not automatically complete the others.

Separate an equipment project from the evaluation purchase

A multiple-candidate application may run on existing equipment. This article does not establish a need for additional GPUs or a physical installation.

If the selected application separately creates an equipment installation requirement, use Leviathan's Complete Physical Infrastructure Deployment service and the contextual project inquiry to discuss the identified equipment, location and IT connections. That is an agreed physical delivery scope. Model evaluation, candidate selection and application-performance acceptance remain with the appointed software provider.