AI System Unlocks Visual Categories in New Contexts

A New Approach to Image Categorization

A groundbreaking method known as Open Ad-hoc Categorization (OAK) is changing how AI systems interpret images. Unlike traditional approaches that rely on fixed categories, OAK allows AI to dynamically reinterpret the same image based on different contexts. This innovation was introduced in a study led by the University of Michigan and presented at the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) in June 2025.

The research challenges the common assumption that each image has a single, objective meaning. Instead, it emphasizes the flexibility of perception, similar to how humans adjust their understanding of an image depending on the situation or goal. Stella Yu, a professor of computer science and engineering at U-M and senior author of the study, explained that AI should be able to adapt its interpretation just like people do.

How OAK Works

Traditional AI categorization systems use rigid categories such as "chair," "car," or "dog." These systems struggle to adapt when faced with different tasks or contexts. OAK, however, can assess the same image in multiple ways. For example, an image of a person drinking could be categorized as "drinking," "in a store," or "happy," depending on the context.

To build their model, the research team expanded upon OpenAI's CLIP, a vision-language AI model that connects images with text. They introduced context tokens—specialized instruction sets that guide the AI to focus on relevant parts of the image. These tokens are trained using both labeled and unlabeled data, allowing the system to adjust its focus without explicit guidance.

Importantly, the original CLIP model remains unchanged during training, ensuring that the system retains existing knowledge while adapting to new tasks. Zilin Wang, a doctoral student at U-M and lead author of the study, noted the effectiveness of this approach, highlighting that even with minimal input, the system can learn to focus on the right areas of an image.

Discovering New Categories

One of the most impressive features of OAK is its ability to discover new categories that were not part of the training data. For instance, when asked to identify items suitable for a garage sale, the system could recognize items like luggage or hats—even if it had only seen examples of shoes during training.

This capability comes from combining top-down and bottom-up approaches. The top-down method uses language knowledge to suggest potential new categories. If the system knows that shoes can be sold at a garage sale, it might infer that hats could also be sold there. The bottom-up method, on the other hand, identifies patterns in unlabeled visual data. For example, if many suitcases appear in images related to garage sales, the system may recognize them as a new category.

During training, these two methods work together. Semantic proposals prompt the system to search for specific items, and if found, they confirm a valid category. Visual clusters help the system label what it discovers using existing knowledge from CLIP.

Performance and Applications

The researchers tested OAK on two image datasets—Stanford and Clevr-4—and compared its performance against baseline models such as CLIP with an extended vocabulary and Generalized Category Discovery (GCD). OAK achieved state-of-the-art results in both accuracy and concept discovery.

Notably, OAK reached 87.4% novel accuracy when identifying mood in the Stanford dataset, outperforming CLIP and GCD by over 50%. Its saliency maps, which highlight relevant image regions, are more flexible and interpretable than those generated by other methods.

Looking ahead, OAK’s contextual approach has potential applications in fields like robotics, where systems need to perceive the same environment differently based on their current task. The research was also supported by the University of California, Berkeley and the Bosch Center for AI.

For more information, visit the official OAK research page.