Balaur: Language Model Pretraining with Lexical Semantic Relations
Résumé
Lexical semantic relations (LSRs) characterize meaning relationships between words and play an important role in systematic generalization on lexical inference tasks.Notably, several tasks that require knowledge of hypernymy still pose a challenge for pretrained language models (LMs) such as BERT, underscoring the need to better align their linguistic behavior with our knowledge of LSRs.In this paper, we propose BALAUR, a model that addresses this challenge by modeling LSRs directly in the LM's hidden states throughout pretraining.Motivating our approach is the hypothesis that the internal representations of LMs can provide an interface to their observable linguistic behavior, and that by controlling one we can influence the other.We validate our hypothesis and demonstrate that BALAUR generally improves the performance of large transformer-based LMs on a comprehensive set of hypernymyinformed tasks, as well as on the original LM objective.Code and data are made available at github.com/mirandrom/balaur.
Citer ce document
Exporter : BibTeX · RIS (Zotero, Mendeley, EndNote)
Accès au document
Texte intégral archivé sur SPHAERO, en accès ouvert : lisible gratuitement en ligne après connexion. Le document reste aussi disponible à sa source. Se connecter
Voir l'article sur le site de la revueLicence et provenance
Licence : CC BY-SA
Notice moissonnée depuis OpenAlex le 10/09/2026. Le document reste hébergé par sa source.
Voir le document à la source →