Systematic polysemy in adjective-noun combination in contextual word embeddings
Michael Goodale, Salvador Mascarenhas
October 2023 

In this work, we find evidence that masked language models such as BERT vary their representations of adjectives in a manner that reflects non-trivial theoretical predictions made by linguistic semantic theories about how adjectives compose with nouns. For example, “skilled surgeon” and “skilled carpenter” are skilled in different ways, whereas a “French surgeon” and “French carpenter” are French in the same way. Crucially, we demonstrate via a simple probe that the variation of the adjective depends on the noun in question for adjectives like “skilled” and the noun is not useful for predicting the variation of adjectives like “French”. We also show that this variation is systematic—it extends to novel nouns unseen in training. These novel nouns are generated by a process we call “nonce-embeddings”, a technique which samples novel embeddings from the manifold of embeddings of nouns in order to generate “meaningless” words akin to nonce-words like wug or blicket used in linguistics and psychology.

Format:	[ pdf ]
Reference:	lingbuzz/007644 (please use that when you cite this article)
Published in:
keywords:	word embeddings, adjectives, subsective, intersective, intensional, probing, language models, bert, distributional semantics, polysemy, semantics
Downloaded:	257 times

 

[ edit this article | back to article list ]