IoU-PD: IoU-Aware Privileged Distillation for Visual Grounding with Multimodal Large Language Models

Ground-truth boxes guide a privileged teacher during training, while the deployed student keeps the standard image-and-text grounding interface.

Xiuyuan Zhu^1,2 Ke Lu^1,3 Hao Wu^1,2 Siwen Jiao⁴ Zijin Du¹ Dongming Zhang² Jian Xue^1,*

¹ University of Chinese Academy of Sciences

² State Key Laboratory of Communication Content Cognition

³ Peng Cheng Laboratory

⁴ National University of Singapore

arXiv Paper PDF

Model

BibTeX Contact

Privileged information is used only during training One student, two training views

Student-only inference

Student Original input

Image + referring expression

Frozen teacher Privileged input

Box-marked image + location hint

SFT anchor Privileged KL distillation IoU-aware token weights

Inference Original image + prompt → student → bounding box No box overlay, teacher branch, or additional prediction module

Abstract

Visual grounding with multimodal large language models is commonly formulated as autoregressive coordinate generation, where a model outputs bounding-box coordinates as text given an image and a referring-expression prompt. While this interface is simple and compatible with instruction following, it introduces a mismatch between training and evaluation: training optimizes token-level likelihood over coordinate strings, whereas grounding quality is measured by geometric overlap. We propose IoU-PD, an IoU-aware privileged distillation method for coordinate-generating multimodal large language models. IoU-PD uses ground-truth boxes not only as coordinate targets, but also as privileged training-time guidance. During training, the student receives the original image and prompt, while a frozen teacher receives a box-marked image and an augmented prompt that indicates the marked region. The student is trained with a supervised fine-tuning anchor and a privileged distillation loss whose token weights reflect both geometric importance and teacher reliability. At inference time, IoU-PD requires no box overlay, privileged hint, teacher branch, or additional prediction module. Experiments on standard referring-expression grounding benchmarks show consistent region-level improvements over strong coordinate-generating baselines, demonstrating that ground-truth boxes can provide useful privileged guidance beyond serving as coordinate labels.

Training should reflect geometry

Coordinate generation is trained as next-token prediction, but evaluated by the overlap between two regions. IoU-PD introduces geometric information into training while preserving the generative format.

Coordinate SFT label only

Image + query Coordinate string

The ground-truth box supplies the output tokens. Token likelihood does not directly describe the geometric quality of the predicted region.

IoU-PD label + privileged guidance

Coordinate target Box-marked teacher view Geometry-aware token weights

The same box also guides a frozen teacher. Distillation then transfers this training-only information to the student according to geometric importance and teacher reliability.

Privileged training viewThe teacher sees a box-marked image and a short hint.

Extra inference modulesDeployment uses the original student input and output.

Held-out evaluation splitsRefCOCO, RefCOCOg, and RefCOCO+ benchmarks.

IoU-aware privileged distillation

IoU-PD combines direct coordinate supervision with a frozen privileged teacher and token-level weighting that connects distillation to region quality.

Privileged teacher

The teacher receives a bounding-box overlay and a short hint while the student receives the original input.

SFT anchor

Supervised coordinate generation maintains the output format and anchors the grounding distribution.

IoU-aware weighting

Sample quality, coordinate error, digit significance, agreement, and confidence determine token weights.

IoU-PD framework with an original-input student, a frozen box-marked teacher, supervised fine-tuning, and IoU-aware privileged distillation — The student learns from the original input. A frozen teacher receives privileged box guidance during training. Only the student is retained for inference.

Consistent region-level gains

Under the matched Qwen3-VL-4B setting, IoU-PD improves the pooled result and every held-out split across mIoU, Acc@0.5, and Acc@0.7.

OVERALL+4.03mIoU points

OVERALL+2.98Acc@0.5 points

OVERALL+4.25Acc@0.7 points

COVERAGE5 / 5splits improved on all three metrics

Matched 4B comparison Base → IoU-PD

Evaluation split	mIoU		Acc@0.5		Acc@0.7
Evaluation split	Base	IoU-PD	Base	IoU-PD	Base	IoU-PD
Overall	81.74	85.78	88.58	91.56	82.51	86.76
RefCOCO testA	85.90	88.45	93.25	95.19	88.56	91.44
RefCOCO testB	81.45	84.20	88.85	90.95	81.33	84.14
RefCOCOg test	81.85	87.23	88.31	91.45	82.18	87.34
RefCOCO+ testA	83.79	87.14	90.85	93.59	85.98	89.91
RefCOCO+ testB	74.64	79.88	80.73	85.80	73.33	79.28

Values are percentages. Overall results pool all examples from the five held-out splits.

Scaling ablations across data sizes, epoch budgets, and final model settings — The improvement remains visible with reduced data and model size, and increases with additional grounding data and a sufficient training budget.

Where the gains come from

Component, threshold, and object-size analyses show that the improvement is distributed across the evaluated conditions rather than confined to one metric or subset.

Component ablations for supervised fine-tuning, teacher distillation, privileged box input, and IoU-aware weighting — SFT provides the main anchor. The privileged box teacher and IoU-aware weighting add further gains.

Grounding precision across IoU thresholds and the predicted IoU distribution — IoU-PD improves performance from P@0.5 through P@0.95 and shifts predictions toward higher overlap.

Base and IoU-PD precision for small, medium, and large target objects — Both P@0.5 and P@0.7 improve for small, medium, and large objects.

Inside the training signal

The visual hint preserves the scene while identifying the target. The token-weighting rule then allocates the distillation signal according to geometry and confidence.

Comparison of bounding-box, Gaussian blur, reverse-shadow, and grayscale privileged visual hints — A bounding-box overlay marks the target while retaining the scene context needed by the referring expression.

Token weights for coordinate digits based on coordinate error, digit significance, teacher agreement, and confidence — Token weights vary across coordinates and digit positions, concentrating distillation where it is most useful.

Citation

If this work supports your research, please cite the paper.

@article{zhu2026ioupd,
  title   = {IoU-PD: IoU-Aware Privileged Distillation for Visual Grounding with Multimodal Large Language Models},
  author  = {Zhu, Xiuyuan and Lu, Ke and Wu, Hao and Jiao, Siwen and Du, Zijin and Zhang, Dongming and Xue, Jian},
  journal = {arXiv preprint arXiv:2607.15732},
  year    = {2026}
}

Contact

For questions about the paper, code, or model, please contact us by email.

Email zhuxiuyuan22@mails.ucas.edu.cn