A model can connect three words creatively because the words already identify the concepts to connect. Three images demand an additional step: discovering which concepts matter.
CREATE's October 13 paper uses visual Remote Associates Test problems, with human validation and evaluations of eleven vision-language models. Its findings suggest that associative strength depends on textual cues and does not transfer uniformly to visual puzzles. Read the research paper.
Verbal association can outsource its hardest first step
Creative language tasks often supply the concepts to connect. Three words already identify candidate meanings, leaving the model to search for an association. Three images add another problem: deciding which visual concepts matter before linking them. A fluent visual explanation may conceal that dependence if a caption or prompt has already selected the intended concepts.
CREATE evaluates this boundary through visual Remote Associates Test problems. Each item asks for a connection among three seemingly dissimilar images. The benchmark does not try to measure artistic merit or creativity in all its forms. It isolates a particular kind of lateral association in a visual setting.
Human validation anchors a deliberately small benchmark
The researchers semi-automatically construct one hundred and fifty-five problems and use a human study with twenty-five participants to validate and ground the benchmark. They then evaluate eleven vision-language models. Most fall below human performance in the reported results. The experimental comparisons suggest that associative ability depends on textual cue concepts and does not transfer uniformly to image-only puzzles.
The image sources also shape the task. The authors use free stock-photo collections rather than synthetic imagery, limiting puzzles to concepts with suitable available photographs. This avoids some image-generation quality concerns but narrows the range and number of problems. The resulting benchmark is a controlled sample, not a comprehensive map of visual imagination.
Captions can change what is being evaluated
Supplying a concept label may remove ambiguity that constitutes part of the original challenge. If a model succeeds after receiving those labels, it may be demonstrating a strong associative step while still struggling with visual concept selection. Paired image-only and text-assisted conditions make that division visible instead of crediting the entire chain to one undifferentiated creative ability.
The paper also notes that some connections depend on specialised senses of otherwise familiar words. An incorrect answer can reflect missing vocabulary or domain knowledge as well as difficulty using known concepts creatively. The experiment does not fully distinguish those causes. That limitation is important when interpreting a gap as a general creativity deficit.
A familiar word can hide an unfamiliar conceptual bridge
The limitations give an example involving driver, shell and pipe linked through operating system. The words are familiar, but the intended senses require particular domain knowledge. A model can fail because it does not activate that sense, because it cannot identify the relevant visual concept or because it cannot connect concepts it already knows. Those are different deficiencies.
This suggests a useful extension for creative-assistant evaluation: pair puzzles with checks of vocabulary and concept recognition before measuring the final association. Such checks would not replace the creative task; they would clarify which prerequisite failed. CREATE’s current result establishes a boundary in the tested visual association setting while leaving that more detailed causal decomposition for further work.
A design assistant should reveal where the leap occurred
In practical ideation, human-supplied labels can be entirely appropriate. The issue is attribution, not whether assistance is allowed. If the user or captioning pipeline supplies the key abstraction, the model's contribution is narrower than a claim that it discovered the connection directly from the scene. Evaluation should preserve that boundary.
Visual creativity requires choosing a useful representation before searching for an association. A system can be impressively inventive once concepts are named while remaining unreliable at choosing the relevant concepts from images. CREATE offers a way to inspect that gap, helping developers decide whether to improve perception, concept selection, knowledge or the associative reasoning that follows.
References
Questions, news tips or corrections? Contact [email protected].
