Accès ouvert · CC BY

Adaptive Hard Negative Mining for Fine Grained Vision Language Understanding A Negative Result and Analysis

Article scientifique 2026 Sciences mathématiques et statistiques Anglais

Résumé

Abstract Vision-language models perform well on many multimodal tasks but struggle with fine-grained visual-linguistic reasoning. The SPEC benchmark was introduced to evaluate models on four fine-grained concepts: object size, position, existence, and count. Fine-tuning on SPEC with fixed hard negatives improves baseline performance substantially. This paper asks whether adaptive hard-negative mining, which reweights negatives by their current difficulty, can push performance further. We propose a confusion-weighted training objective and evaluate it on SPEC held-out data. The method underperforms fixed hard-negative training by 23 percentage points on average. We trace the cause to SPEC's design: its candidate sets are uniformly hard, so adaptive reweighting has no difficulty signal to exploit and instead destabilizes training. We also introduce SPEC++, a paired-reasoning diagnostic that combines cases from two SPEC subsets. The fine-tuned model's joint accuracy on SPEC + + equals the product of its individual accuracies, indicating that it treats concepts independently rather than composing them. These two findings serve as cautionary notes for the field: uniform hard negatives are difficult to outperform when the data is designed to be uniformly hard, and high benchmark accuracy should not be read as evidence of compositional understanding.

Citer ce document

Ayitey, S., & Owusu-Agyemang, K. (2026). Adaptive Hard Negative Mining for Fine Grained Vision Language Understanding A Negative Result and Analysis. Research Square. https://doi.org/10.21203/rs.3.rs-11030933/v1

Exporter : BibTeX · RIS (Zotero, Mendeley, EndNote)

Accès au document

Texte intégral archivé sur SPHAERO, en accès ouvert : lisible gratuitement en ligne après connexion. Le document reste aussi disponible à sa source. Se connecter

Voir l'article sur le site de la revue

Licence et provenance

Licence : CC BY

Notice moissonnée depuis OpenAlex le 09/11/2026. Le document reste hébergé par sa source.
Voir le document à la source →

Statistiques

Consultations (hors robots) : 6

Téléchargements (hors robots) : 0

Méthode de comptage