A kitchen scene can suggest a hidden kettle from familiar context. A shadow or reflection may identify an occluded object much more specifically—and prove harder for a vision model.
OccludeBench's October 9 paper separates direct physical cues from broader inference. Evaluated models show a substantial gap between those categories. A diagnostic pipeline locates difficulty recovering an object from its projection within the full image. Read the research paper.
Scene familiarity can disguise weak physical inference
A hidden object can be inferred in two very different ways. Familiar context may suggest what is normally present, while a reflection or shadow can constrain what must be present in this particular scene. A vision-language model can succeed at the first through broad scene knowledge without mastering the second. Conventional recognition scores can blur those abilities because the likely answer and the physically supported answer often coincide.
OccludeBench separates them using fully occluded objects. Its direct cases require a specific physical cue; its inferential cases permit recovery from broader context. That distinction asks whether the visible evidence actually constrains the model's hypothesis about what cannot be seen directly.
The difficult cases carry more specific evidence
Across five evaluated vision-language models, the authors report average accuracy of 42.7 percent on direct cases and 62.7 percent on inferential cases, a twenty-point gap. Human accuracies are 63.4 and 68.5 percent, respectively, without a significant gap. The human comparison does not make the benchmark trivial; it shows that the model's relative difficulty differs from the human pattern.
The surprising result is that more specific physical evidence can be less useful to the model than contextual expectation. A system may know what belongs in a room while struggling to recover an object from the projection that reveals its presence. For physical agents, those are materially different kinds of knowledge.
A diagnostic locates the projection bottleneck
The four-stage OccludeTrace procedure investigates where the reasoning breaks down. In recovering the relevant object from its projection within the full image, the models succeed on only half of examples they ultimately fail, compared with ninety-one percent of examples they answer correctly. Longer reasoning, tool use and few-shot prompting provide inconsistent benefits and do not reliably remove the category gap.
An inpaint-then-confront intervention takes another route: generate a candidate hidden object, then check that hypothesis against the physical cue. For the two evaluated models with image-generation capabilities, the category gap becomes statistically non-significant. That is a result for a specific intervention and sample, not permission to treat generated content as observed evidence.
Noticing the shadow is not the missing step
OccludeTrace finds cue detection at ninety-nine percent on failed examples and one hundred percent on successful ones. Models usually notice that a shadow or reflection exists. Cropping the projection also yields relatively strong object recognition, and source mapping does not distinguish the failed group cleanly. The largest separation appears when the model must recover the depicted object from the marked region within the intact scene.
That pattern is more specific than saying the models cannot see shadows. It suggests that a locally readable cue can be lost when interpreted in the whole image. The proposed remedy should therefore be evaluated for whether it changes attention to the relevant physical content, rather than merely adding a longer verbal explanation of an already detected cue.
A hypothesis must answer to the image
The intervention's logic matters more than the act of imagining. A proposed object is useful only if the remaining cue can confirm or challenge it. Without confrontation, generation could reinforce the same familiar-scene guess the benchmark is designed to expose. Evaluations should keep the physical consistency check visible.
For robotics and spatial assistants, tests of shadows, reflections and hidden occupancy should complement ordinary object recognition. A plausible scene description can be insufficient when an action depends on whether an unseen obstacle is really there. A model may reason fluently about a scene while missing the physical constraint that makes it different from a familiar example.
Questions, news tips or corrections? Contact [email protected].
