CooKie: commonsense knowledge-guided mixture-of-experts framework for fine-grained visual question answering

In the FG-VQA task, VQA models need to answer questions about the detailed visual content of images using open-world knowledge. Recently, MLLMs based on large language models have shown powerful multimodal performance across diverse tasks including FG-VQA. Nonetheless, those end-to-end MLLMs' generalization performance remains limited due to visual information loss caused by a lack of flexibility (i.e. one-offtask-agnostic visual perception), especially in high-resolution scenarios. As an alternative solution, another series of methods utilize an LLM-based tool-using system, adopting flexible strategies and calling expert models to handle various situations. However, these solutions still achieve unsatisfactory improvement due to the following limitations: 1) Simplistic task decomposition ignoring the potential failure of sub-tasks, and 2) Over-liance on the expert models' limited ability. To this end, we extend the use of LLM-based system and propose a simple but effective framework called CooKie (COmmOnsense Knowled-ge-uided MIxture-of-Experts), which leverages the inherent knowledge of an MLLM as guidance for conducting visual searches in a manner similar to human behavior. In addition to this region-aware search strategy, we introduce a robust post-hoc selection module with a Mixture-of-xperts strategy, integrating another expert model to provide a supplementary reference. In the experiments, CooKie showcases a new state-of-the-art (SOTA) performance of 79.06% accuracy on V* Bench. Meanwhile, it achieves about 3% improvement on POPE and GQA with comparative performance on MME-Bench. This indicates that our method specializes in high-resolution FG-VQA tasks as well as object hallucination reduction and maintains the general multimodal capability. Furthermore, we manually construct additional instances as the expansion of V* Bench to better evaluate the generalization performance of our method, demonstrating CooKie's effectiveness.