Accès ouvert · CC BY-SA

Balaur: Language Model Pretraining with Lexical Semantic Relations

Article scientifique 2023 Sciences mathématiques et statistiques Anglais

Résumé

Lexical semantic relations (LSRs) characterize meaning relationships between words and play an important role in systematic generalization on lexical inference tasks.Notably, several tasks that require knowledge of hypernymy still pose a challenge for pretrained language models (LMs) such as BERT, underscoring the need to better align their linguistic behavior with our knowledge of LSRs.In this paper, we propose BALAUR, a model that addresses this challenge by modeling LSRs directly in the LM's hidden states throughout pretraining.Motivating our approach is the hypothesis that the internal representations of LMs can provide an interface to their observable linguistic behavior, and that by controlling one we can influence the other.We validate our hypothesis and demonstrate that BALAUR generally improves the performance of large transformer-based LMs on a comprehensive set of hypernymyinformed tasks, as well as on the original LM objective.Code and data are made available at github.com/mirandrom/balaur.

Citer ce document

Mircéa, A., & Cheung, J. C. K. (2023). Balaur: Language Model Pretraining with Lexical Semantic Relations. https://doi.org/10.18653/v1/2023.findings-emnlp.674

Exporter : BibTeX · RIS (Zotero, Mendeley, EndNote)

Accès au document

Texte intégral archivé sur SPHAERO, en accès ouvert : lisible gratuitement en ligne après connexion. Le document reste aussi disponible à sa source. Se connecter

Voir l'article sur le site de la revue

Licence et provenance

Licence : CC BY-SA

Notice moissonnée depuis OpenAlex le 10/09/2026. Le document reste hébergé par sa source.
Voir le document à la source →

Statistiques

Consultations (hors robots) : 9

Téléchargements (hors robots) : 0

Méthode de comptage