Real-Time Evaluation: Measuring Decision Quality Under Latency Constraints

The challenge: the world keeps moving

A space robot approaching a rotating target must make decisions against a changing geometric relationship. The useful lifetime of an observation depends on relative motion, pose uncertainty, actuator response, and the distance remaining before a maneuver becomes irreversible. A detailed plan generated from an old image can be less useful than a simpler action based on a fresh state estimate. For embodied intelligence, reasoning therefore consumes a resource that is also part of the task: time.

This report develops an evaluation design in which the environment continues evolving during inference. Its purpose is to make a practical distinction: understanding a task, choosing an appropriate action, and delivering that action before its deadline are three separate requirements. Model selection should expose all three. A benchmark that pauses the environment while the model deliberates answers a different engineering question from one that includes observation age and command delivery.

Paper brief: STAR makes timing observable

STAR compares turn-based and real-time decision-making in a partially observed, zero-sum game. Four layers separate an ECS engine, environment rules, asynchronous protocol, and agents. PWER extends Elo with resource-preservation and completion-time terms. Tables 1–2 report Kimi-K2-Thinking at 1206.1 PWER in turn-based play and 842.6 in real time; Qwen3-30B-A3B-Instruct scores 877.3 and 1156.7. These are benchmark-specific ratings. The comparison motivates timing-aware evaluation rather than treating reasoning quality as an isolated scalar.

Engineering design: instrument the complete loop

Instrument five events for every decision: sensor exposure, observation availability, inference start, command acceptance, and actuator response. Retain these timestamps even when a command fails validation. Otherwise an apparently fast agent can obtain a misleading advantage by emitting invalid actions that never reach the plant. Separate model compute from queueing, preprocessing, communication, and actuator delay so that an optimization can be traced to the subsystem it actually improves.

Define action age as the interval from the observation that informed a command to the time its physical effect begins. This quantity is often more useful than tokens per second. A cache can improve throughput while delivering older observations; batching can reduce average cost while increasing the tail delay of an urgent command. Recording action age reveals both effects. A decision record should also include the state-estimator revision, model revision, tool arguments, validation result, and the trajectory segment actually applied.

Engineering design: separate planning from commitment

A useful architecture has a persistent planner and a short-period execution supervisor. The planner proposes a goal or a motion primitive; the supervisor checks whether its assumptions still hold at dispatch. Each proposal carries an expiry time, a coordinate frame, required state predicates, and an interruption condition. An expired plan is a recorded decision outcome, not an excuse to freeze the simulation. This makes stale decisions visible and keeps high-level language generation from silently defining the control clock.

For an inspection task, the slow process may choose the next surface region while a fast process maintains a permitted standoff distance. During capture preparation, the slow process may compare approach sequences while the fast process preserves a retreat corridor. These are proposed task decompositions: they let a benchmark evaluate planning depth and response speed without forcing one model to operate at every time scale. Compare them against a single-loop baseline using the same sensing and actuation budgets.

Evaluation protocol: paired clocks, shared scenarios

Run each scenario in two modes. In deliberation mode, advance the simulator only after a valid decision; this characterizes planning under a generous time budget. In continuous mode, advance simulation independently and apply accepted commands at their actual arrival times. Use the same initial states and disturbance seeds. Report changes within each model across the two modes as well as differences between models, because otherwise an easier scenario sample can be mistaken for better temporal behavior.

Sweep latency deliberately rather than observing only one server configuration. Add controlled delays to image delivery, model service, and command transport separately. Include bursty delays, because the same mean latency can conceal very different worst-case behavior. Pair every run with an event log and replayable state trace. For resource comparisons, fix a power or memory envelope and report successful tasks per unit energy alongside task success. This identifies whether additional reasoning earns its operating cost.

Metrics: do not hide failure behind an average

Use a scorecard rather than a single ranking. Include task completion, final pose error, minimum separation, deadline-miss rate, action-age percentiles, invalid-command rate, energy, and recovery duration. Distinguish a safe timeout from an unsafe command, and distinguish an infeasible plan from one that became infeasible while waiting. The latter is specifically a temporal failure. Keep these outcomes separate even when an executive dashboard eventually combines them into a utility score.

An engineering utility can reward completion while subtracting normalized energy and deadline costs, with unacceptable constraint violations treated as explicit exclusions. Publish the normalization scales and weights. A model should not rise in rank merely because the designer chose an arbitrary energy unit. Where outcomes are stochastic, use paired confidence intervals over scenario seeds and show the distribution of failure times. A few late catastrophic failures deserve visibility even when the median episode looks excellent.

Experiment matrix: measure the value of extra thought

A compact study can compare a fast instruction model, a reasoning model, and a hierarchical combination. Hold the observation format and allowed tools constant. Then vary target angular speed, observation noise, and the decision deadline. This produces a response surface instead of an isolated leaderboard. The most informative boundary is where extra thought ceases to improve the executed outcome. At that boundary, a scheduler can shorten reasoning, request a new observation, or choose a bounded fallback primitive.

The deliverable should be a reproducible decision trace: what the robot knew, when it knew it, what it proposed, and what happened after execution. That trace ties model behavior to physical consequences and makes latency optimization reviewable. The central design lesson is operational: evaluate the action at the moment it reaches the world. For space embodied systems, timely intelligence becomes an observable system property rather than a promise inferred from static answers.