Model Peer Review: Calibrated Scoring for Candidate Selection

The challenge: a plausible plan is not enough

A space robot can receive a high-level instruction that admits several plausible execution sequences. Inspecting a component may require changing viewpoint, managing illumination, preserving a communication opportunity, and keeping a recovery path available. Different plans can satisfy the wording while imposing very different costs on the physical system. Selecting a plan therefore requires more than fluency: it requires comparing alternatives against shared evidence and a precise definition of task completion.

An ensemble is useful when it exposes distinct hypotheses and makes selection reviewable. The important artifact is not a longer conversation between models, but a structured set of proposals, objections, and decisions. This report develops an engineering workflow for high-level planning in which model review is paired with deterministic checks. The workflow preserves why a proposal was accepted, which alternatives were rejected, and which assumptions need to be checked again before execution.

Paper brief: score, infer, select

LLM-PeerReview selects among model-generated candidates. Flipped triples balance scoring order; aggregation uses either averaging or a latent score model with judge-specific confusion matrices and EM estimation. The final choice maximizes aggregated score. In v4, Table 1 reports averages of 67.4 and 67.8 versus Smoothie-Global's 60.5 across TriviaQA, GSM8K, MATH, and AlpacaEval. These mixed-task averages are reported benchmark measures. The engineering sections below propose a mission-planning integration.

Engineering design: define a candidate contract

Require each candidate to use a common schema: objective, ordered actions, preconditions, expected observations, estimated resource use, abort conditions, and success predicates. References to objects and frames must resolve against the same state snapshot. This prevents a reviewer from awarding a high score to an eloquent plan that quietly assumes a different target pose or an unavailable tool. Preserve the original candidate unchanged after scoring so that selection and subsequent execution remain traceable.

Candidate diversity should be deliberate. One proposal can prioritize observation quality, another minimize maneuver cost, and another preserve a larger retreat margin. These are different optimization preferences over the same task, not invitations to invent missing state information. Compare the proposed action sequences after normalization; cosmetic paraphrases should not count as independent options. If several models converge on the same sequence, retain that fact without pretending that the candidate set covers multiple strategies.

Engineering design: separate physical checks from review

Run executable checks before asking reviewers to express preferences. Validate object identifiers, action schemas, joint limits, collision queries, available energy, and ordering constraints. A failed hard check should be an explicit rejection with a machine-readable reason. Reviewers can then compare the remaining candidates for instruction coverage, assumption clarity, observation sufficiency, and recoverability. This division gives each subsystem a narrow responsibility and prevents a persuasive explanation from compensating for a known infeasible action.

Give reviewers the same evidence bundle and a fixed rubric. Hide candidate model names and vary presentation order when estimating order sensitivity. Require a cited action identifier for every objection so the objection can be checked against the plan. A useful review says that action A3 depends on an observation omitted by A2; an ungrounded preference for a confident writing style is not useful operational evidence. Store both the scalar score and the structured objection.

Engineering analysis: understand disagreement

A score matrix contains two kinds of information: estimated candidate quality and disagreement about that quality. A consistently severe reviewer may still rank candidates correctly; a generous reviewer may miss the most important distinction. Examine calibration on a separate set of plans with independently checked defects. Track performance by defect type, such as missing observation, wrong frame, resource overrun, or invalid action order. This is more informative than a single global label of strong reviewer.

For a simple deployment baseline, average normalized rubric scores only after hard feasibility filtering. Retain the score spread and the gap between the top two candidates. A small gap combined with substantial disagreement should trigger targeted review of the disputed action rather than automatic expansion into a long debate. The equation is a proposed engineering selection rule: Q is the rubric score, w is the chosen criterion weight, and F is the feasible candidate set.

Resource design: spend review effort where it matters

Use staged review. Begin with a compact candidate set and inexpensive structural checks. Add another reviewer only if a decision remains ambiguous on a criterion that can change the outcome. Cache fixed mission constraints and static geometry checks, but invalidate results whenever the relevant state changes. Separate candidate generation cost from review cost and aggregation cost. This prevents the term lightweight from hiding expensive upstream calls that dominate the actual operating budget.

A mission-level plan can be reviewed before a narrow execution window begins, while short-period control remains under an independent supervisor. Assign an expiry condition to the selected plan and recheck only affected assumptions when new observations arrive. A new target pose need not force reviewers to repeat their analysis of an unchanged mission requirement. This dependency-aware design turns review into a maintainable computation graph rather than a repeated full-document judgment.

Evaluation protocol: measure selection, not eloquence

Build test sets in which each mission has several feasible candidates and carefully inserted defects. Freeze the candidate pool when comparing selection methods. Measure chosen-plan success, defect detection, unnecessary rejection, review latency, energy, and total model calls. Also calculate selection regret relative to the best candidate in the fixed pool under simulator evaluation. This separates failure to generate a good plan from failure to recognize a good plan that was already available.

Ablate reviewer count, candidate diversity, ordering, rubric detail, and the physical-check stage. Keep the final execution evaluator independent from the reviewers. For every accepted plan, retain a compact decision packet containing the input state, candidates, checks, scores, objections, selected identifier, and resulting execution trace. This packet is the main engineering deliverable. It makes model collaboration a documented decision process and supports precise diagnosis when a future mission exposes an overlooked assumption.