Evaluating the Existence Proof: LLMs as Cognitive Models of Language Acquisition
Hector Javier Vazquez Martinez, Annika Heuser, Charles Yang, Jordan Kodner
July 2024 

In recent years, the technological success of large language models (LLMs) has been taken as an existence proof that language acquisition may succeed without domain-specific principles and constraints. While this argument acknowledges the important differences between LLM training and child language acquisition, its validity rests on the validity of the existence proof itself, that LLMs indeed demonstrate capacity comparable to human linguistic knowledge, the terminal state of the acquisition process. We contend that such a proof has not been delivered, in large part due to the lack of rigor in LLM evaluation and the absence of serious engagement with the empirical study of child language. When trained on child-scale input data and evaluated on widely used benchmarks, LLMs can be readily matched by simple baseline models that are demonstrably inadequate for human language. As a partial remedy, we advocate for the use of thoroughly validated datasets that more accurately reflect the scope of linguistic knowledge. On these datasets, even LLMs trained on very large amounts of data perform in a way inconsistent with human behavior. The burden of an existence proof is considerably heavier than previously realized.

Format:	[ pdf ]
Reference:	lingbuzz/008277 (please use that when you cite this article)
Published in:	Forthcoming. In Artificial Knowledge of Language. José-Luis Mendívil-Giró, editor. Vernon Press
keywords:	computational linguistics, cognitive modeling, language acquisition, llm, llms, neural networks, benchmarking, evaluation, methodology, syntax, syntax
Downloaded:	1073 times

 

[ edit this article | back to article list ]