Every time I'm not reading, I'm thinking about reading.
Title | Category | Medium | Tags |
|---|---|---|---|
Landmark Paper. First to prove that AI models like BERT secretly store a massive database of facts via their weights. This changed AI history (before the rise of daily LLM use) to a digital brain that knows everything.
Main Exploration: How much relational knowledge do they store? How does this differ for different types of knowledge such as facts about entities, common sense, and general question answering? How does their performance without fine-tuning compare to symbolic knowledge bases automatically extracted from text?
Here, LAMA consists of a massive set of "fill-in-the-blank" (cloze) questions from high-quality sources. Here, LAMA would have to fill in the "masked" section, such as "Dante was born in [MASK]" without being allowed to search.We introduce the LAMA (LAnguage Model Analysis) probe, consisting of a set of knowledge sources, each comprised of a set of facts.
Causal/Autoregressive (GPT style): Models only look backwards to predict the next word at the end of the sentence.
Cloze/Bi-directional (BERT style): Models look at words before and after the blank to get an understanding of context and relational facts.
Unidirectional (Causal): Models used a "masked future". While they learn, they are "blindfolded" to everything that comes after the current word.
Bidirectional (Cloze): models have "full visibility". They can see the whole sentence at once, which makes them better at understanding the relationship between words.
This paper showed that
Note that because this paper was in 2019, some information may have been improved upon in more recent research.