Entities › Artificial systems

Large language models

Transformer neural networks trained to predict the next token in very large text corpora, then often tuned on human feedback.

Transformer neural networks trained to predict the next token in very large text corpora, then often tuned on human feedback. They use self-attention over the whole context window. One 2020 paper reported a scale of 175 billion parameters. They produce fluent language, so human-sounding output is a poor test of inner experience.

Sources

• Vaswani A et al. (2017). Attention is all you need. NeurIPS. arXiv:1706.03762.
• Brown TB et al. (2020). Language models are few-shot learners. NeurIPS 33. arXiv:2005.14165 (175B parameters).

Assess it against the published criteria