---
title: "Large language models · Artificial systems · Entities · Simulation Hypothesis"
url: https://simulation-hypothesis.com/entities/ai-llm
description: "Transformer neural networks trained to predict the next token in very large text corpora, then often tuned on human feedback."
license: https://creativecommons.org/licenses/by/4.0/
credit: "Ross Allan Dahl, Simulation Hypothesis (simulation-hypothesis.com)"
---

[Entities](https://simulation-hypothesis.com/entities) › Artificial systems

# Large language models

Transformer neural networks trained to predict the next token in very large text corpora, then often tuned on human feedback.

Transformer neural networks trained to predict the next token in very large text corpora, then often tuned on human feedback. They use self-attention over the whole context window. One 2020 paper reported a scale of 175 billion parameters. They produce fluent language, so human-sounding output is a poor test of inner experience.

In the trees

[Transformer architecture · Technology](https://simulation-hypothesis.com/trees/technology?n=transformer)[Generative agents · Simulations](https://simulation-hypothesis.com/trees/simulations?n=min-park)[Indicators of AI awareness · Ethics](https://simulation-hypothesis.com/trees/ethics?n=c-butlin)[Taking AI welfare seriously · Ethics](https://simulation-hypothesis.com/trees/ethics?n=c-long)

## Sources

* Vaswani A et al. (2017). Attention is all you need. NeurIPS. arXiv:1706.03762.
* Brown TB et al. (2020). Language models are few-shot learners. NeurIPS 33\. arXiv:2005.14165 (175B parameters).

Assess it against the published criteria