Prof. Rabovsky is not so much interested in the structures and rules of language. “Language is structured by syntax. But its real purpose is to convey meaning,” she says. In her research, she uses so-called neural network models, also known as deep learning models. These are computer models that, among other things, are supposed to reproduce how the human brain processes language. She uses them to try to reproduce and explain brain signals measured during human language processing. “If this is successful, it indicates that they might be good models.”
What is special is that the understanding of language underlying neural network models differs significantly from classical linguistic theories of language processing. These theories assume that the functionality of language is based on syntactic rules. They also often assume that these cannot be learned, but that basal aspects of these syntactic rules are innate. The idea behind this is a universal grammar, as described by the linguist Noam Chomsky, for example. For neuroscientist Rabovsky, the role of syntactic rules is no longer of primary importance. For the task of language to convey meaning, statistical regularities are often more essential and, above all, more effective than syntactic rules. That is why many people initially misunderstand the sentence “The dog was bitten by the man”. They understand what they expect, namely that the dog bites the man, and not, as the syntax suggests, that the dog is bitten by the man.
Learning language in use
“Neural network models learn language from the statistical regularities of the environment,” Rabovsky explains, and in different ways, depending on the approach. “My model, for example, learns to map linguistic input regarding its meaning by estimating – based on sequentially presented words – what event each sentence describes, i.e. who is doing what to whom (or what), and getting feedback on these estimations to improve them.” Other models try to predict the next word in each sentence. “The models learn language simply by performing their task.” Syntactic rules are also important for this. After all, the texts with which the models learn and train language comprehension contain normal and thus regular sentences. The rules are not anchored in the models in advance though. Rather, they learn them as they use the language. “So, the way I see language processing is very different from the way many linguists understand it,” she says. “And I think the impressive power of current deep learning models of language processing (e.g. chatGPT) at least challenges the linguistic idea that language cannot be learned without some prior knowledge.”
Prof. Rabovsky’s models ultimately learn language the same way humans do: They listen, over and over again. To be exact, they are “fed” with so-called corpora, collections of typical printed texts, often consisting of newspaper editions. At some point the models recognize – or rather, they master – regularities. At this point, it gets really interesting for Rabovsky, since her research actually focuses on those linguistic phenomena that deviate from the norm. Like the sentence about the biting man and the poor dog. The neuroscientist is researching the N400, which is neither a supersonic aircraft nor a kitchen machine, but a waveform. The N400 wave can be measured in humans with the help of electroencephalography (EEG) and is a component of the so-called event-related potential. The wave provides information about our speech processing. Or more precisely: It indicates when our brain is having difficulties with it. The N400 is triggered whenever words appear that do not fit the context. A classic example: “I take my coffee with cream and dog.” That dog again. The waveform gets its name from the delay with which our brain registers the deviation: It is triggered 400 milliseconds after we read or hear “dog”. The signal, i.e. the wave, peaks the more the word deviates from the expectation. If we drink our coffee with honey, it is less noticeable than with dog.
Human and model in the experiment
Rabovsky has worked with the N400 for a long time because the trained psychologist “grew up” in EEG research. There, the wave is one of the most important instruments for investigating language processing. After all, it reliably shows how our brain deals with linguistic phenomena that it is not (yet) familiar with. “We have been able to demonstrate N400 effects in numerous studies,” she says. “But for a long time, it was not clear what exactly was behind it, what the brain actually does when the N400 amplitude gets higher or lower. That was pretty frustrating.” In search of new approaches, the cognitive scientist turned to modelling, went to Canada to the University of Western Ontario and later to Stanford. Since 2019, she has been at the University of Potsdam. Here, thanks to funding from the German Research Foundation (DFG) within the Emmy Noether Program, she has established a group bringing together experimental research and modelling – currently primarily focusing on the N400. “There is a theory that the N400 indicates a prediction error and the associated learning effect,” she explains. When we hear speech, the brain is constantly trying to predict the next word and its meaning. If there is a discrepancy, there is an error message, the N400 wave peaks and the brain has to correct its prediction. It learns.
“We have implemented these assumptions in our models and are now working on testing them,” she says. After the models were trained with corpora for some time, they are now compared with humans. To do this, Rabovsky and her team basically conduct two experiments in parallel: In the EEG lab, test subjects are confronted with sentences that repeatedly contain deviations of varying degrees. Sometimes there is milk with honey, sometimes with dog. Or something similar. The researchers then compare the recorded N400 waves with the activation patterns of the models that were “fed” with the same linguistic input. “If we can detect the same N400 effects in the models as in our test subjects, this is an indication that they could be good models of human language processing,” Rabovsky says. They also want to investigate the learning effect experimentally: A typical N400 test is followed by a second “learning test” to show whether the size of the N400 in the first part predicts later learning effects. “If the N400 is indeed a learning signal, subjects should recognize those words faster in the second trial where they showed a higher N400 amplitude.” The newly learned becomes an old dog.
Ideally, this explanation can then also be verified with the help of the model. So far it has passed all the endurance tests. Rabovsky is satisfied and is already turning to new goals. She is, for example, planning an MEG study together with the Max Planck Institute for Human Cognitive and Brain Sciences. Thanks to the magnetoencephalography (MEG) used in this study, the researchers can no longer just record when something happens in the brain – as with the EEG – but even where.
Learning from mistakes
In addition, a second wave is to be integrated into the model in the future: The P600. “While the N400 reflects something like the automatic process of error detection, the P600 possibly shows reanalysis,” Rabovsky explains. “This means a relatively controlled cognitive process in which people correct the error and correctly grasp what has been misunderstood.” Who misunderstands, subsequently corrects and then correctly grasps the meaning should be shown by the two waves. Since this process is definitely a cognitive challenge, the experiments also have to take various factors into account. Whether the test subjects are attentive, for example. In the future, not only EEG but also eye movement measurements will be used for this – and, of course, the “grown” model.
The model, in any case, seems convincing to Rabovsky. In her view, even if it is currently a highly simplified model, the multitude of explained neuronal effects suggests that the principles underlying the model capture essential aspects of human language processing. And although she emphasizes that she is conducting basic research that has the task of investigating human language comprehension, she already sees potential applications. “If you have a really good model, you can use it to better understand aphasias, for example, or to explain mechanisms of diseases that affect our faculty of language.”
There is still a long way to go, but the close cooperation with colleagues at the Institute of Linguistics in Potsdam is particularly inspiring in this respect. There are no reservations between linguistics and cognitive neurosciences. On the contrary: Prof. Rabovsky works very productively with some of them in the linguistic collaborative research center “Limits of Variability in Language”. “We have similar interests and it fits very well on the experimental side. Besides, we are all tolerant. And maybe everyone secretly thinks that, in the end, they will convince the others.”
The Researcher
Prof. Dr. Milena Rabovsky studied psychology at Humboldt-Universität zu Berlin. After she did her PhD at the Berlin School of Mind and Brain. After research stays in London (Ontario), Stanford, and Freie Universität Berlin, she has been Professor for Cognitive Neuroscience at the University of Potsdam since 2019.
Email: milena.rabovskyuuni-potsdampde
The Project
Combining electrophysiology and deep neural network models in research on language comprehension (Emmy Noether Junior Research Group)
Funding: German Research Foundation (DFG)
Duration: 2019–2024
Participants: Prof. Dr. Milena Rabovsky (group leader), Professor Dr. James McClelland (Stanford University)
What deep learning models of language processing can do is demonstrated by Open AI’s chatGPT, which was launched at the end of 2022. It is freely available to all and, based on input, is able to generate all kinds of texts – from romantic poems to scientific essays.
https://openai.com/blog/chatgpt
This text was published in the university magazine Portal Wissen - One 2023 „Learning“ (PDF).