Scaling Test-Time Robustness of Vision-Language Models via Self-Critical Inference Framework

The emergence of Large Language Models (LLMs) has driven rapid progress in multi-modal learning, particularly in the development of Large Vision-Language Models (LVLMs). However, existing LVLM training paradigms place excessive reliance on the LLM component, giving rise to two critical robustness challenges: language bias and language sensitivity. To address both issues simultaneously, we propose a novel Self-Critical Inference (SCI) framework that extends Visual Contrastive Decoding by conducting multi-round counterfactual reasoning through both textual and visual perturbations. This process further introduces a new strategy for improving robustness by scaling the number of counterfactual rounds. Moreover, we also observe that failure cases of LVLMs differ significantly across models, indicating that fixed robustness benchmarks may not be able to capture the true reliability of LVLMs. To this end, we propose the Dynamic Robustness Benchmark (DRBench), a model-specific evaluation framework targeting both language bias and sensitivity issues. Extensive experiments show that SCI consistently outperforms baseline methods on DRBench, and that increasing the number of inference rounds further boosts robustness beyond existing single-step counterfactual reasoning methods.

• SCI, a counterfactual inference framework that simultaneously mitigates language bias and enforces language consistency.
• DRBench, a model-specific and dynamic benchmark that assesses LVLM robustness on samples from real downstream tasks.
• SCI consistently improves performance on both DRBench and standard datasets, and the paper reveals a previously underexplored way to improve robustness: increasing the number of test-time counterfactual inference rounds.

Visual Contrastive Decoding (VCD) runs a second inference with a noisy image and decodes from the contrasted logits. Expanding its softmax shows that VCD simply reweights the original token probability by exp(TIE/τ), where TIE, the Total Indirect Effect introduced by Counterfactual VQA (CF-VQA) and Unbiased Scene Graph Generation, is the logits with the real image minus the logits with the noisy image. The extra parameter α of VCD is a temperature, τ = 1/α. Object hallucination is therefore treated as iterative biased token generation, and an LVLM is fundamentally no different from a conventional VQA classifier at every decoding step.

VCD only handles the visual side. SCI adds a Textual Counterfactual (TC) component that makes the logits prompt-consistent, and strengthens the Visual Counterfactual (VC) component with several counterfactual images:

• v⁰ and q⁰ are the original image and prompt. qⁱ are semantically equivalent but lexically different prompts. v^j are content-removed images.
• TC takes the element-wise maximum over the N+1 prompt variants on every vocabulary dimension. VC averages over M counterfactual images to obtain a more stable estimate than a single TIE.
• Following VCD, an Adaptive Plausibility Constraint masks low-confidence tokens before decoding. SCI uses the temperature-scaled TC logits as the masking criterion, because TC provides more consistent predictions than the original logits.
• Special cases. VCD is SCI with N=0 and M=1. CF-VQA is SCI with a constant TC term and M=1.
• SCI3, SCI5, SCI7 denote M+N+1 = 3, 5, 7 total inference rounds, that is M=N=1, 2, 3.

Existing test-time scaling increases the length of intermediate thinking tokens within a single inference. SCI scales a different axis: the number of counterfactual inference rounds whose logits are aggregated. Overall accuracy on all three DRBench subsets rises from the base model (1 round) to SCI3, SCI5 and SCI7.

Fixed robustness datasets are costly to build, can leak into the web-crawled training data of later models, and often use a single question type such as exist-or-not questions. More importantly, non-robust samples are not fixed: 24.68% of the test samples are hard for LLaVA-NeXT, yet only 7.34% are shared with Qwen2-VL. DRBench therefore builds the benchmark per model, in two steps.

• Evaluate. Run the model on an existing dataset with the original input, two visual counterfactual inputs and two textual counterfactual inputs (M=N=2).
• Filter. The Bias Subset collects samples that yield the same incorrect prediction under both the original and the dummy visual inputs, which indicates reliance on spurious language priors. The Sensitivity Subset collects samples whose prediction changes in response to subtle, non-causal prompt variations. The BS Subset is their union.

The paper constructs DRBench from six widely used benchmarks: MME, MMStar, CCBench, ViLP, MMBench-DEV-EN-V11 and MMBench-DEV-CN-V11, randomly split into 20% validation (3,315 samples) and 80% test (13,251 samples: 10,632 multiple-choice and 2,619 others). Results are reported for MCQ (multiple-choice questions) and Others (Yes/No for MME, open-ended QA for ViLP).

Base models: the Hugging Face versions of Qwen2-VL-7B-Instruct (bfloat16, default top-k sampling) and Llama3-LLaVA-NeXT-8B (float16, greedy decoding). Baselines: TIE adapted from CF-VQA, VCD, and M3ID, all with the adaptive plausibility constraint. All experiments use VLMEvalKit on a single NVIDIA A800 80GB GPU. The metric is top-1 accuracy (%).

The repository is a modified copy of VLMEvalKit v0.2. SCI, TIE, VCD and M3ID are implemented for Qwen2-VL and LLaVA-NeXT at the logit level inside the model's forward, and every variant is a registered model name in vlmeval/config.py, for example Qwen2-VL-7B-SCI5-b02a1g2t03.