LEC talk Friday 23 October: Jelle Zuidema

By jon | October 13, 2015

Friday 23 October, 16:00–17:30
Room 1.17, DSB

Jelle Zuidema, Institute for Logic, Language and Computation, University of Amsterdam

Deep learning models of language processing and the evolution of syntax

Researchers that assume discrete, symbolic models of syntax, tend to reject gradualist accounts in favor of a ‘saltationist’ view, where the crucial machinery for learning and processing syntactic structure emerges in a single step (as a side-effect of known or unknown brain principles or as a macro-mutation). A key argument for the last position is that the vast amount of work on describing the syntax of different natural languages makes use of symbolic formalisms like minimalist grammar, HPSG, categorical grammar etc., and that it is difficult to imagine process of any sort that moves gradually from a state without to a state with their key components. The most often repeated version of this argument concerns ‘recursion’ or ‘merge’, which is said to be an all or nothing phenomenon where you can’t have just a little bit of it.

In my talk I will review this argument in the light of recent developments in computational linguistics where a class of models has become popular that we can call ‘vector grammars’, that include the Long-Short Term Memory networks, Gated Recurrent Networks and Recursive Neural Networks (e.g., Socher et al., 2010). These models have over the last two years swept to prominence in many areas of NLP and shown state of the art results on many benchmark tests including constituency parsing, dependency parsing, sentiment analysis and question classification. They have in common that they are much more expressive than older generation of connectionist models (roughly speaking, as expressive as grammar formalisms popular in theoretical linguistics), but replace the discrete categorical states from those formalisms with points in a continuously valued vector space and hence quite easily allow accounts of gradual emergence (as exploited when these models are trained on data using variants of backpropagation).

I will conclude that these new type of models, although mostly evaluated in a language engineering context, hold great promise for cognitive science and language evolution. For theories of the evolution of grammar and syntax they effectively undermine a key argument that the only explanatory models are incompatible with gradualist evolution; rather, they show that syntactic category membership, recursion, agreement, long-distance depencies, hierarchical structure etc. can all gradually emerge, both in learning and in evolution.