RESEARCH / TECHNICAL REPORTS
← Technical reportsRESEARCH NOTE / 06

Explainable perception

Patchwise Attribution: Tracing Answers to Visual Evidence

Read the paper ↗GitHub ↗

Estimate how image patches contribute to a vision-language model’s answer.

AstroUp ResearchTACL, volume 13, pp. 744–759 · 2025Technical analysis
Uncertain visual evidence

Changing illumination, occlusion and surface appearance make visual evidence difficult to trust.

Patch attribution · visual stress testing

THE CENTRAL IDEA

For a space robot, a useful answer should come with evidence that survives controlled intervention. Separate model faithfulness, human plausibility, and action validation.

Patchwise Attribution: Tracing Answers to Visual Evidence

The challenge: an answer must identify its evidence

A space manipulator approaching a service panel must distinguish a graspable handle from a cable, a port from a reflection, and an accessible feature from one hidden behind another component. A vision-language model can express the scene in convenient language, but that language alone leaves a critical engineering question unanswered: which visual information actually supported the decision? When the next operation changes contact state, a plausible sentence is a weak substitute for knowing whether the correct interface was perceived.

This motivates an evidence layer between semantic understanding and task execution. The layer should answer a precise question for a fixed image, instruction, and candidate response: which regions increase the model’s support for that response? Patchwise Cooperative provides a concrete way to ask that question. Its contribution is especially useful for reviewing failed inspections, comparing perception models, and deciding which observations deserve closer examination before a plan is accepted. These are distinct uses of the same attribution signal, with different latency requirements.

A cooperative game over image patches

The paper divides an image into a 12 × 12 grid. Each patch becomes a player; a coalition is a subset of visible patches. For a fixed text input T and answer A, the game’s utility is the model’s generation likelihood P(A | I, T). The contribution of patch i is assessed by comparing a coalition with and without that patch. Averaging marginal contributions over coalition contexts produces a Shapley attribution. This matters because two individually ambiguous regions may become informative together: contribution is evaluated in context, rather than inferred from brightness or a single attention coefficient.

The following expression is the standard Shapley formulation of this mechanism. C is the set of patches, N its size, and u is the answer-likelihood utility. Positive attribution means that revealing a region tends to support the chosen answer across coalition contexts. It is therefore necessary to record the answer being explained alongside the heatmap. A heatmap for “the port is visible” and one for “the port is obstructed” need not identify the same evidence, even when they are computed from an identical camera frame.

φᵢ = Σ[S⊆C∖{i}] |S|!(N−|S|−1)! / N! · [u(S∪{i})−u(S)]
01Image + question
02Patch-mask queries
03Local surrogate
04Shapley map
05Intervention review
Paper mechanism: patch-level coalition queries feed a local surrogate; engineering review adds task-linked evidence checks.

Local approximation makes the game tractable

Enumerating all subsets of 144 patches is impractical. The paper instead queries the large model on 1,000 binary patch masks for one input and fits a small two-layer perceptron to the resulting answer probabilities. A binary mask indicates which patches are retained; the surrogate learns the relationship between visibility patterns and response likelihood. Its squared-error objective is local to that image–question–answer combination. Monte Carlo sampling then estimates patch contributions using the cheaper surrogate rather than repeatedly executing the full vision-language model for every coalition.

This is a useful distinction for deployment planning: the lightweight network is an explanation surrogate, not a compressed replacement for the perception system. In a proposed review pipeline, the fitting stage should retain a held-out set of masks, report probability-fit error, and preserve the mask seed for reproducibility. A visually appealing map with poor local fit should be treated differently from a map whose surrogate reproduces the queried model closely. These additional checks make the approximation visible to engineers rather than hiding it inside a visualization.

minθ Eₘ[(Pθ(m) − P(A | Iᵐ,T))²]

Faithfulness and plausibility are different tests

The paper evaluates faithfulness with ROAD perturbation tests. Most Relevant First removes regions ranked important; Least Relevant First removes low-ranked regions. A useful attribution should produce a strong likelihood decline in the first case and a smaller change in the second. This intervention-based test is more informative than asking whether a highlighted region simply looks reasonable. The evaluation perturbs selected pixels using their neighbors rather than relying only on a conspicuous blank rectangle, reducing the extent to which the removal operation itself dominates the result.

For plausibility, the authors collect 6,000 annotations from 60 participants and retain 891 images with aggregated human patch selections. TID1 measures the distance from the model’s highest-ranked patch to the nearest of three human-selected patches; TID3 uses the nearest pair among both top-three sets. These are spatial proximity measures, not overlap scores or proof that all three selections agree. On LLaVA-v1.5-7B, the reported TID1/TID3 values are 0.905/0.387, compared with 3.642/1.632 for LVLM-Interpret. Keeping the metric definition beside the numbers prevents an overly broad reading of the improvement.

Reading model behavior across layers and questions

The experiments cover LLaVA-v1.5 and ShareGPT4V in 7B and 13B sizes, with results averaged over three random seeds. The paper also compares question paraphrases and image styles, and inspects explanations derived from intermediate layers. These analyses make attribution a diagnostic instrument: it can show where evidence becomes concentrated as a response develops. The layer examples suggest that a correct response and concentrated target evidence can emerge together, while some earlier responses rely on more dispersed information.

For engineering review, concentration should remain a descriptive signal. Consider a question asking whether a collision corridor is empty: broad image coverage may be exactly what that question requires. Conversely, a sharply focused map can still point to the wrong connector. A proposed task-aware review should therefore distinguish object identification, object absence, attribute recognition, and clearance assessment. Each question family needs a suitable expected evidence structure. This avoids turning a useful diagnostic pattern into a universal acceptance rule.

Engineering mapping: an evidence packet for each decision

A proposed space-robot implementation can package the camera frame, crop coordinates, instruction, candidate answer, model version, attribution map, and perturbation curve as one evidence record. The record should be linked to the relevant target instance and planned operation. If a grasp proposal refers to a latch, an operator should be able to inspect both the attributed region and the geometric latch estimate without manually aligning unrelated logs. This turns explainability into a traceable interface between semantic perception and the existing robotics stack.

A practical integration can use two timescales. The operational perception loop produces detections and estimates at its normal cadence. A separate analysis service processes selected frames after confidence drops, model disagreement, operator requests, or task failures. Attribution results can then guide a closer view, a request for a different question, or offline data curation. These are proposed control-interface choices: thresholds should be determined from task validation, and the physical safety supervisor continues to evaluate geometry, contact constraints, and reachable motion.

A concrete evaluation plan

Build a review set around interfaces that matter to the mission: grasp points, fasteners, cables, ports, damage indicators, and exclusion regions. For each frame, collect a short task-specific question and independent region annotations. Compare answer quality before comparing heatmaps. Then evaluate deletion curves, attribution stability under paraphrase, sensitivity to crop scale, and the relationship between explanation quality and downstream decision errors. Split data by hardware instance and viewing sequence so that repeated frames do not inflate the apparent generalization of the explanation pipeline.

Measure explanation latency separately from perception latency, along with large-model query count, surrogate fitting cost, peak memory, and review throughput. Retain failures where the answer is right for the wrong region and where the answer is wrong despite plausible evidence. Those cases are valuable for selecting new data and refining task prompts. The engineering outcome is a repeatable evidence workflow: semantic claims become inspectable, failures become attributable, and model changes can be judged by the visual information they actually use.

SOURCE PAPER & REFERENCE MATERIAL

Patchwise Cooperative Game-based Interpretability Method for Large Vision-language Models

Yao Zhu, Yunjian Zhang, Zizhe Wang, Xiu Yan, Peng Sun, Xiangyang Ji

TACL, volume 13, pp. 744–759 · 2025 · Zizhe Wang: Tsinghua University

Continue to the next report →