# Self-Critical Inference (SCI) and DRBench

> Official project of the CVPR 2026 paper "Scaling Test-Time Robustness of Vision-Language Models via Self-Critical Inference Framework" (arXiv:2603.07659) by Kaihua Tang, Jiaxin Qi, Jinli Ou, Yuhua Zheng, and Jianqiang Huang. SCI is a decoding-time framework that runs multi-round textual and visual counterfactual inference and aggregates next-token logits to mitigate language bias (including object hallucination) and language sensitivity in Large Vision-Language Models (LVLMs). DRBench is a model-specific Dynamic Robustness Benchmark.

Key facts (all numbers are from the paper, arXiv:2603.07659v2):

- SCI formulation: p_SCI(y | v, q) ∝ exp(TC/τ1) · exp(VC/τ2), where TC_k = max_i Z_k(v0, q_i) over the original prompt and N semantically equivalent prompt variants, and VC = Z(v0, q0) − E_j[Z(v_j, q0)] over M content-removed images. An adaptive plausibility constraint masks low-confidence tokens.
- Visual Contrastive Decoding (VCD) is the special case N=0, M=1. Counterfactual VQA (CF-VQA, TIE) is the special case with a constant TC term and M=1. The paper shows VCD equals reweighting the original token probability by exp(TIE/τ) with τ = 1/α.
- SCI3, SCI5, SCI7 use M+N+1 = 3, 5, 7 inference rounds. Robustness increases with the number of counterfactual rounds (a test-time scaling direction based on rounds rather than longer reasoning chains).
- Counterfactual inputs: VC-Color0 (black image), VC-Noise500 and VC-Noise400 (diffusion forward noise), TC-V1 (instruction to focus on image details), TC-V2 (instruction language switched between English and Chinese), TC-V3 (identity prompt: a smart student).
- Hyperparameters: τ1 = 1.5 / 2.0 / 2.5 for SCI3 / SCI5 / SCI7, τ2 = 0.2, plausibility threshold β = 0.3 on DRBench and 0.8 on the original datasets.
- DRBench: for a given LVLM, the Bias Subset contains samples with the same incorrect prediction under the original and content-removed images; the Sensitivity Subset contains samples whose prediction changes under equivalent prompts; the BS Subset is their union. Built from MME, MMStar, CCBench, ViLP, MMBench-DEV-EN-V11, MMBench-DEV-CN-V11 (20% validation = 3,315 samples, 80% test = 13,251 samples).
- Main result, BS Subset overall top-1 accuracy: LLaVA-NeXT-8B 18.75 (base), 27.31 (TIE), 27.89 (VCD), 29.05 (M3ID), 32.72 (SCI3), 34.19 (SCI5), 34.92 (SCI7). Qwen2-VL-7B 14.52 (base), 22.32 (TIE), 23.12 (VCD), 25.68 (M3ID), 26.94 (SCI3), 29.50 (SCI5), 31.72 (SCI7).
- Original six datasets, overall accuracy: LLaVA-NeXT 70.46 → 70.79 with SCI5; Qwen2-VL 80.58 → 80.98 with SCI5.
- Non-robust samples are model-specific: 24.68% of test samples are hard for LLaVA-NeXT and only 7.34% are shared with Qwen2-VL. On the BS Subset built by LLaVA-NeXT, LLaVA-NeXT scores 18.75 while Qwen2-VL scores 60.31.
- Overhead with batch inference: 1.29× (SCI3), 1.81× (SCI5), 2.48× (SCI7) of the base inference time; 2.96× / 5.01× / 6.68× when rounds run sequentially (MMStar, Qwen2-VL, one NVIDIA A800 GPU).
- Base models: Qwen2-VL-7B-Instruct and Llama3-LLaVA-NeXT-8B (Hugging Face versions). Code is built on VLMEvalKit v0.2. License: Apache-2.0.

## Paper

- [arXiv abstract](https://arxiv.org/abs/2603.07659): abstract, authors, versions
- [arXiv PDF](https://arxiv.org/pdf/2603.07659): full paper with appendix
- [arXiv HTML](https://arxiv.org/html/2603.07659v2): full text in HTML

## Code

- [GitHub repository](https://github.com/KaihuaTang/Self-Critical-Inference-Framework): implementation of SCI, TIE, VCD, M3ID and DRBench construction, with reproduction steps
- [README](https://github.com/KaihuaTang/Self-Critical-Inference-Framework/blob/main/README.md): method summary, paper-to-code map, full result tables
- [ViLP.tsv for VLMEvalKit](https://github.com/KaihuaTang/Custom-Dataset-for-VLMEvalKit): dataset file required by the pipeline

## Project page

- [English](https://kaihuatang.github.io/Self-Critical-Inference-Framework/): method, charts, all result tables, FAQ
- [中文](https://kaihuatang.github.io/Self-Critical-Inference-Framework/zh/): Chinese version

## Citation

```bibtex
@inproceedings{tang2026scaling,
  title={Scaling Test-Time Robustness of Vision-Language Models via Self-Critical Inference Framework},
  author={Tang, Kaihua and Qi, Jiaxin and Ou, Jinli and Zheng, Yuhua and Huang, Jianqiang},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2026}
}
```