Joint State–Action Diffusion: Generating Controllable Test Scenarios

The challenge: rare interactions dominate test value

A space manipulation campaign needs more than a large collection of ordinary approach trajectories. The informative cases occur when uncertain target motion, limited visibility, actuator limits, and tight timing interact. Sampling each variable independently may create many easy cases and a few impossible ones. Neither efficiently reveals the boundary of an autonomous system's competence. Test generation must therefore search for combinations that remain physically meaningful while exposing a specific weakness.

The engineering objective is controlled difficulty. A test author should be able to ask for a shorter observation window, a smaller clearance, or a larger recovery demand without losing the ability to replay the resulting episode. This makes generative modeling useful beyond image appearance: the generated object is an evolving interaction that includes the controls responsible for motion. The following design connects that representation to a space robot simulation workflow.

Paper brief: joint state–action denoising

E2E-CDiff jointly generates future states and controls. Its state head consumes detached predicted actions during denoising; guidance restores action-to-state gradients. Objectives control speed, admissible regions, and collision interaction. Bench2Drive experiments use CARLA 0.9.15, 220 training routes and 10 test routes. With PlanT as ego planner, Table II reports 2D-TTC 2.92 ± 1.53 and speed Wasserstein distance 3.83 ± 0.73. The following sections develop an independent spacecraft-oriented engineering design around joint generation.

Engineering design: choose a physical representation

Represent an episode by time-aligned observations, states, and commands. For a free-floating manipulation scenario, a candidate state can contain base pose, base twist, joint positions, joint velocities, target pose, and target twist. A command can contain commanded joint torques and permitted base-control inputs. Store frame conventions and units beside every field. Rotations require a representation with an explicit normalization or manifold projection; an unnormalized orientation vector can create an attractive but invalid sample.

Normalize variables using meaningful engineering scales rather than a single statistical standard deviation across the entire state. A millimeter of clearance and a degree of orientation error have different operational significance. The representation should retain contact mode and actuator availability because two geometrically similar trajectories can require different control actions. When a variable is discrete, such as a gripper contact state, model or validate its transition separately instead of treating every intermediate numeric value as physically meaningful.

Engineering design: connect generated motion to dynamics

A generated trajectory is a proposal. Replay its commands through the simulator from the specified initial state, and compare the simulated rollout with the generated state sequence. The residual measures internal consistency: does applying the proposed control actually produce the proposed future? Weight translational, rotational, and joint errors separately. Retain the full residual trace so that an initially plausible sample that drifts later in the horizon can be identified.

The equation below is an engineering consistency criterion, not a claim about a reported experimental result. F denotes the chosen dynamics integrator and W defines physically meaningful error weights. An acceptance threshold should be tied to the intended test resolution. If collision detection depends on centimeter-scale separation, a much larger state-replay error would make the scenario ambiguous. This check also makes the generator replaceable: different generative models can feed the same physical acceptance pipeline.

Engineering design: steer difficulty with interpretable costs

Construct separate objectives for visibility, clearance, control effort, observation delay, and recovery time. The test author should know which objective produced a difficult episode. A visibility objective can shorten the interval during which a designated feature remains observable; a clearance objective can move an interaction toward a specified separation band; an effort objective can bring a controller near a torque or momentum-management threshold. These objectives define the challenge in measurable quantities instead of relying on a vague label such as hard.

Distinguish the simulator's hard feasibility checks from the generator's soft preferences. Increasing a difficulty weight should not silently allow impossible acceleration or interpenetrating geometry. One practical workflow is to guide proposals toward a challenge region, reject violations of hard constraints, and record the rejection rate. A generator that produces attractive failures only after rejecting nearly all samples has a different operating cost from one that reliably finds valid boundary cases.

Evaluation protocol: coverage before spectacle

Create a scenario ledger indexed by target spin, relative velocity, illumination, pose uncertainty, actuator budget, and latency. Hold out entire combinations rather than only random frames from familiar episodes. Compare a manually authored library, uniform parameter sampling, and guided generation under an equal simulation budget. The comparison should count distinct valid failure modes and the amount of newly covered operating space, not only the largest number of collisions a search can produce.

Separate generator quality from the tested controller's strength. Re-evaluate a fixed scenario set after changing the controller, and re-evaluate a fixed controller after changing the generator. Save initial conditions, random seeds, simulator version, integration settings, and every applied action. A replay test should reproduce the defining event within a stated tolerance. That record turns a visually compelling episode into an engineering artifact that another team can inspect and reproduce.

Implementation roadmap: build a reusable scenario asset

Begin with one tightly defined interaction, such as approach-and-retreat around a rotating body. Build a trusted simulator interface and a compact command vocabulary before increasing generative complexity. The first milestone is a batch of physically consistent episodes with independently adjustable visibility and clearance. The second is a closed-loop test in which the target behavior and robot decisions influence one another. The third is a regression library that preserves every confirmed failure and its repaired controller version.

The useful output is not merely a trajectory file. It is a scenario package containing the intended challenge, the executable controls, the resulting sensor sequence, validity checks, and a concise failure explanation. Joint state–action generation becomes valuable when all of these agree. This supports a disciplined feedback loop: discover a difficult interaction, diagnose the controller response, improve the policy, and rerun the same evidence before expanding the test envelope.