---
title: "Film, media and art · Simulations · Simulation Hypothesis"
url: https://simulation-hypothesis.com/simulations/media-art
description: "Cloth, smoke, crowds and light, generated from physical models rather than animated by hand. Part of a taxonomy of the simulations people actually run."
license: https://creativecommons.org/licenses/by/4.0/
credit: "Ross Allan Dahl, Simulation Hypothesis (simulation-hypothesis.com)"
---

[Simulations](https://simulation-hypothesis.com/simulations)domain 17 of 18

# Film, media and art

Cloth, smoke, crowds and light, generated from physical models rather than animated by hand.

Before computers, film and animation approximated physics with wire rigs, miniatures and hand-drawn frames. Now simulation supplies the physics directly: draping cloth, moving hair, crowds that don't repeat, and light that travels the way it does in the real world. The methods of engineering and science are pointed at a story instead of a bridge or a reactor. They're tuned to look believable, not to be physically exact, and they often run offline, frame by frame, instead of in real time. Music and image share one idea: set up the underlying process and let the result emerge.

## Simulated physical media

### Cloth simulation

Fabric represented as a mesh of points and springs, draping, folding and tearing under gravity and movement.

Question it answers: How does this garment move on a moving body without an animator posing every fold?

Leaves out: Real fabric has a memory of its folds that most cloth solvers ignore.

Runs on: about 10^12 FLOPS rough a render farm node for one shot's cloth pass Since: 1987 by 1987, early mass-spring cloth models for computer animation

[Continuum fields](https://simulation-hypothesis.com/simulations/methods#m-continuum)[Differential equations](https://simulation-hypothesis.com/simulations/methods#m-equations)floor: Fields

### Hair and fur simulation

Tens of thousands of individual strands, each obeying its own bending and collision rules, moved by wind, gravity and the body beneath.

Question it answers: How does a full head of hair or a coat of fur move believably, strand by strand?

Leaves out: Strand-to-strand friction is approximated. Otherwise each simulation would take longer than a production schedule allows.

Runs on: about 10^12 FLOPS rough a render farm node for one character's hair pass Since: 1989 by 1989, early strand-based hair animation research

[Continuum fields](https://simulation-hypothesis.com/simulations/methods#m-continuum)[Differential equations](https://simulation-hypothesis.com/simulations/methods#m-equations)floor: Fields

### Fluid and smoke simulation for effects

Fire, smoke, water, dust and explosions computed as a fluid on a grid or in particles, styled for how they should look rather than for scientific accuracy.

Question it answers: How does this explosion, wave or smoke plume move and look on screen?

Leaves out: Directed to look good in frame, not to conserve energy or mass exactly.

Runs on: about 10^13 FLOPS rough a render farm allocation for one effects-heavy shot Since: 1982 by 1982, early computer-generated fluid and particle effects for film

[Continuum fields](https://simulation-hypothesis.com/simulations/methods#m-continuum)[Particles and N-body](https://simulation-hypothesis.com/simulations/methods#m-particles)floor: Fields

### Destruction and debris simulation

Buildings, vehicles and objects fractured into pieces and thrown apart by rigid-body physics, tuned so the collapse looks dramatic rather than merely correct.

Question it answers: How does this structure break apart, and where do the pieces land?

Leaves out: Real materials fail unpredictably; these fracture patterns are pre-planned to look good.

Runs on: about 10^12 FLOPS rough a render farm node for one destruction shot Since: 1995 by 1995, rigid-body fracture simulation used in film effects

[Particles and N-body](https://simulation-hypothesis.com/simulations/methods#m-particles)[Differential equations](https://simulation-hypothesis.com/simulations/methods#m-equations)floor: Bodies

### Muscle and skin deformation

A digital body's skin and soft tissue sliding, bulging and wrinkling over an animated skeleton and muscles underneath.

Question it answers: Does the body of this creature or character move as though it had real soft tissue?

Leaves out: Skin sliding over bone is still mostly approximated, not properly simulated tissue.

Runs on: about 10^12 FLOPS rough a render farm node for one character's body pass Since: 1999 by 1999, muscle-based deformation rigs for computer-animated characters

[Continuum fields](https://simulation-hypothesis.com/simulations/methods#m-continuum)[Differential equations](https://simulation-hypothesis.com/simulations/methods#m-equations)floor: Fields

### Rigid body dynamics for animation

Solid objects that collide, stack, roll and settle under gravity and friction, used to animate props and secondary motion without hand-keying every bounce.

Question it answers: How does this pile of objects tumble and settle once it is disturbed?

Leaves out: Objects occasionally interpenetrate or jitter in ways no real thing would.

Runs on: about 10^11 FLOPS rough a workstation for one shot's prop simulation Since: 1988 by 1988, rigid-body physics used for computer animation

[Differential equations](https://simulation-hypothesis.com/simulations/methods#m-equations)[Particles and N-body](https://simulation-hypothesis.com/simulations/methods#m-particles)floor: Bodies

### Position-based dynamics

Objects such as cloth and soft bodies move by directly adjusting point positions to meet constraints, which is fast and stable.

Question it answers: How can deformable objects be simulated fast enough for interactive use?

Leaves out: Not physically exact and stiffness depends on step count.

Runs on: about 10^9 FLOPS rough Real-time on one CPU or GPU Since: 2007 Position based dynamics (2007) [source](https://doi.org/10.1016/j.jvcir.2007.01.005)

[Particles and N-body](https://simulation-hypothesis.com/simulations/methods#m-particles)[Real time](https://simulation-hypothesis.com/simulations/methods#m-real-time)floor: Bodies

### Ocean surface synthesis

A water surface generated from a spectrum of waves, used to render open sea with swell and foam.

Question it answers: What does a sea of given wind and depth look like?

Leaves out: It models surface shape, not the full flow below or the breaking of large waves.

Runs on: about 10^12 FLOPS rough a workstation, minutes per frame Since: 1986 by 1986, Fournier and Reeves, A simple model of ocean waves [source](https://doi.org/10.1145/15922.15894)

[Differential equations](https://simulation-hypothesis.com/simulations/methods#m-equations)[Continuum fields](https://simulation-hypothesis.com/simulations/methods#m-continuum)[Stochastic processes](https://simulation-hypothesis.com/simulations/methods#m-stochastic)floor: Fields

### Particle system effects

Fire, sparks, smoke and spray drawn as thousands of short-lived points, each with its own motion and appearance.

Question it answers: How can a fuzzy event be shown as a cloud of independently moving points?

Leaves out: Points do not push on each other, so true fluid behaviour is absent.

Runs on: about 10^10 FLOPS rough a laptop, real time Since: 1983 by 1983, Reeves, Particle systems (used in a wall of fire effect) [source](https://doi.org/10.1145/964967.801167)

[Particles and N-body](https://simulation-hypothesis.com/simulations/methods#m-particles)[Stochastic processes](https://simulation-hypothesis.com/simulations/methods#m-stochastic)floor: Fields

### Snow and granular material simulation

Snow, sand and similar materials simulated as points that carry stress, so they pile, break and clump like the real material.

Question it answers: How does snow pack, fracture and flow when pushed?

Leaves out: Grain-scale ice physics is replaced by an averaged material law.

Runs on: about 10^14 FLOPS rough a render farm, hours per shot Since: 2013 by 2013, Stomakhin and others, A material point method for snow simulation [source](https://doi.org/10.1145/2461912.2461948)

[Particles and N-body](https://simulation-hypothesis.com/simulations/methods#m-particles)[Continuum fields](https://simulation-hypothesis.com/simulations/methods#m-continuum)floor: Fields

## Populated and built worlds

### Digital crowd simulation

Thousands of individual figures, each with simple movement and avoidance rules, filling a stadium, battlefield or street without being animated one by one.

Question it answers: How does a crowd this size move, flee or fight as a mass?

Leaves out: Every figure in the back rows tends to move a little too similarly.

Runs on: about 10^12 FLOPS rough a render farm allocation for one crowd shot Since: 1997 by 1997, agent-based crowd systems used in feature film battle scenes

[Agent-based](https://simulation-hypothesis.com/simulations/methods#m-agent-based)floor: Bodies

### Digital humans and faces

A face's muscles, skin and micro-expressions modelled well enough to animate a synthetic performance that reads as a real person's.

Question it answers: Can this synthetic face convey emotion as convincingly as the real one it is based on?

Leaves out: The eyes and the corners of the mouth remain the most difficult regions to reproduce convincingly.

Runs on: about 10^13 FLOPS rough a render farm allocation for one hero character's face Since: 2001 by 2001, high-resolution facial capture and simulation for film

[Continuum fields](https://simulation-hypothesis.com/simulations/methods#m-continuum)[Differential equations](https://simulation-hypothesis.com/simulations/methods#m-equations)floor: Fields

### Procedural cities and landscapes

Streets, buildings, terrain and vegetation generated by rules and randomness from a small set of parameters, instead of being placed by hand.

Question it answers: How can an entire city or continent be populated with detail that cannot practicably be placed by hand?

Leaves out: Generated cities tend to be statistically plausible but lack any specific history.

Runs on: about 10^12 FLOPS rough a workstation or render farm node for one environment build Since: 1982 by 1982, fractal terrain generation used in computer graphics

[Rules and states](https://simulation-hypothesis.com/simulations/methods#m-rules)[Stochastic processes](https://simulation-hypothesis.com/simulations/methods#m-stochastic)floor: Rules

### Virtual production sets

A computer-generated environment displayed live on large screens behind actors and filmed in camera, with the scene reacting to how the lens moves.

Question it answers: Can the background react to the camera in real time so it does not have to be added afterwards?

Leaves out: The lighting match between the real actor and the screen behind them remains imperfect.

Runs on: about 10^13 FLOPS rough a cluster of gaming GPUs driving the stage screens live Since: 2019 by 2019, large LED-volume virtual production stages in wide use

[Real time](https://simulation-hypothesis.com/simulations/methods#m-real-time)[Continuum fields](https://simulation-hypothesis.com/simulations/methods#m-continuum)floor: Fields

### Digital ecosystems and vegetation

Forests, grasslands and animal populations generated and animated together, with plants competing for light and animals moving through the result.

Question it answers: How does a whole simulated landscape of plants and creatures look and move together?

Leaves out: Simulated ecosystems are tuned to look lush, not to balance the way real ones do.

Runs on: about 10^12 FLOPS rough a render farm node for one environment's ecosystem pass Since: 2000 by 2000, procedural vegetation and ecosystem tools for film environments

[Agent-based](https://simulation-hypothesis.com/simulations/methods#m-agent-based)[Rules and states](https://simulation-hypothesis.com/simulations/methods#m-rules)floor: Bodies

### Atmospheric and sky simulation

Clouds, fog, rain and sky colour generated from physical models of light scattering and moisture, for a scene's weather rather than the real thing.

Question it answers: What should the sky and weather in this shot look like, without shooting on the right day?

Leaves out: Tuned for how a sky should look, not for genuine atmospheric physics.

Runs on: about 10^12 FLOPS rough a render farm node for one environment's sky pass Since: 1986 by 1986, physically based sky and cloud rendering models

[Continuum fields](https://simulation-hypothesis.com/simulations/methods#m-continuum)[Rules and states](https://simulation-hypothesis.com/simulations/methods#m-rules)floor: Fields

### Neural avatars

A neural network trained on multi-camera captures of a person renders a lifelike, animatable face or body.

Question it answers: Can a virtual person be rendered indistinguishably from the real one?

Leaves out: Needs capture of each subject, and behaviour is not generated.

Runs on: about 10^13 FLOPS rough Per-person training Since: 2018 Deep appearance models for face rendering (2018) [source](https://arxiv.org/abs/1808.00362)

[Learned emulators](https://simulation-hypothesis.com/simulations/methods#m-learned)floor: Bodies

### Virtual idols and influencers

A fictional performer rendered in computer graphics and managed as a celebrity, appearing in media and social feeds.

Question it answers: Does an audience attach to a performer who has no body?

Leaves out: The persona is written by a team; no autonomous inner life is modelled.

Runs on: about 10^11 FLOPS rough a workstation, rendering in near real time Since: 1996 by 1996, Kyoko Date, a 3D CG idol debuted by Horipro [source](https://en.wikipedia.org/wiki/Kyoko%5FDate)

[Real time](https://simulation-hypothesis.com/simulations/methods#m-real-time)[Rules and states](https://simulation-hypothesis.com/simulations/methods#m-rules)floor: Bodies

### Virtual concerts

A live performance staged in a shared virtual world or game engine, with avatars, stage effects and a remote audience.

Question it answers: Can a concert be shared by an audience that is only present as avatars?

Leaves out: Audience reaction and crowd presence are thin compared with a real venue.

Runs on: about 10^12 FLOPS rough a data centre, real time for thousands of viewers Since: 2006 by 2006, Duran Duran performed in Second Life [source](https://en.wikipedia.org/wiki/Virtual%5Fconcert)

[Real time](https://simulation-hypothesis.com/simulations/methods#m-real-time)[Agent-based](https://simulation-hypothesis.com/simulations/methods#m-agent-based)floor: Populations

### Digital de-ageing

A performer's face is rebuilt in 3D and retargeted to a younger or older age, driven by the actor's own acting.

Question it answers: Can a face be shown at another age while keeping the actor's expression?

Leaves out: The result is an appearance model; no ageing biology is simulated.

Runs on: about 10^13 FLOPS rough a render farm, hours per shot Since: 2008 by 2008, The Curious Case of Benjamin Button, with Digital Domain effects [source](https://en.wikipedia.org/wiki/The%5FCurious%5FCase%5Fof%5FBenjamin%5FButton%5F%28film%29)

[Real time](https://simulation-hypothesis.com/simulations/methods#m-real-time)[Continuum fields](https://simulation-hypothesis.com/simulations/methods#m-continuum)[Learned emulators](https://simulation-hypothesis.com/simulations/methods#m-learned)floor: Bodies

### Performance-captured digital characters

A computer-generated character animated by an actor's recorded body and facial movement, composited into live footage.

Question it answers: Can an actor's movement drive a creature that looks nothing like a person?

Leaves out: Muscle and thought are not modelled, only the captured motion.

Runs on: about 10^11 FLOPS rough a workstation, hours per shot Since: 2002 by 2002, Gollum in The Lord of the Rings: The Two Towers [source](https://en.wikipedia.org/wiki/Motion%5Fcapture)

[Real time](https://simulation-hypothesis.com/simulations/methods#m-real-time)[Rules and states](https://simulation-hypothesis.com/simulations/methods#m-rules)floor: Bodies

### L-system plant modelling

Plant shapes grown by repeatedly rewriting a string of symbols into branching structure, then drawn as trees and leaves.

Question it answers: What simple rewriting rules produce a given branching shape?

Leaves out: The rules fit shapes; the chemistry and light competition that drive real growth are absent.

Runs on: about 10^8 FLOPS rough a laptop, seconds Since: 1968 by 1968, Lindenmayer, Mathematical models for cellular interactions in development [source](https://doi.org/10.1016/0022-5193%2868%2990079-9)

[Rules and states](https://simulation-hypothesis.com/simulations/methods#m-rules)[Cellular automata](https://simulation-hypothesis.com/simulations/methods#m-cellular-automata)floor: Cells

### Machinima

Films shot by directing characters and cameras inside a real-time game engine instead of a physical camera or an offline renderer.

Question it answers: What can be filmed in a world that exists only in a game engine?

Leaves out: The engine decides what can be shown; acting is limited to what its controls allow.

Runs on: about 10^11 FLOPS rough a gaming PC, real time Since: 1996 by 1996, Diary of a Camper, a short made in the Quake engine [source](https://en.wikipedia.org/wiki/Machinima)

[Real time](https://simulation-hypothesis.com/simulations/methods#m-real-time)[Rules and states](https://simulation-hypothesis.com/simulations/methods#m-rules)floor: Rules

### Animatronic figures

Mechanical, motor-driven figures of people and animals that perform scripted movement and speech.

Question it answers: How convincingly can a machine act like a living creature?

Leaves out: Motion is replayed from a recording; there is no sensing or decision.

Runs on: no computer. a mechanical controller, no computation Since: 1963 by 1963, Disney's Audio-Animatronics, the Enchanted Tiki Birds [source](https://en.wikipedia.org/wiki/Animatronics)

[Physical analogue](https://simulation-hypothesis.com/simulations/methods#m-analogue)[Rules and states](https://simulation-hypothesis.com/simulations/methods#m-rules)floor: Bodies

### Physics-based character control

A virtual body with joints and muscles is trained to move, jump or fight by reinforcement learning in a physics engine, copying motion-capture clips.

Question it answers: Can a character balance and move naturally, reacting to pushes, without hand-made animation?

Leaves out: Style comes only from the clips it copies, and rare motions and fine hand-work are poorly handled.

Runs on: about 10^15 FLOPS rough days on a GPU workstation per skill. Since: 2018 Peng and colleagues' DeepMimic, example-guided deep reinforcement learning of physics-based character skills. [source](https://doi.org/10.1145/3197517.3201311)

[Learned emulators](https://simulation-hypothesis.com/simulations/methods#m-learned)[Real time](https://simulation-hypothesis.com/simulations/methods#m-real-time)[Differential equations](https://simulation-hypothesis.com/simulations/methods#m-equations)floor: Bodies

## Light and sound

### Physically based light transport

Light rays traced backward from the camera, bouncing off and through every surface, to compute a physically consistent image.

Question it answers: How does light behave in this scene, bounce by bounce, rather than being approximated?

Leaves out: Converges slowly, so noise is still suppressed by heuristics rather than by taking more samples.

Runs on: about 10^14 FLOPS rough a render farm allocation across many nodes for one frame Since: 1986 Kajiya's 1986 path tracing algorithm

[Monte Carlo](https://simulation-hypothesis.com/simulations/methods#m-monte-carlo)[Particles and N-body](https://simulation-hypothesis.com/simulations/methods#m-particles)floor: Fields

### Sound propagation and acoustics

Sound waves traced or modelled as they reflect, absorb and diffract through a virtual room, city street or battlefield, for film and game audio.

Question it answers: How should this space actually sound, given its shape and the materials in it?

Leaves out: Real ears and rooms have idiosyncrasies that acoustic models routinely idealise.

Runs on: about 10^11 FLOPS rough a workstation for one scene's acoustic pass Since: 1968 by 1968, ray-based room acoustics modelling

[Continuum fields](https://simulation-hypothesis.com/simulations/methods#m-continuum)[Particles and N-body](https://simulation-hypothesis.com/simulations/methods#m-particles)floor: Fields

### Physical modelling synthesis of instruments

A string, reed, drum skin or pipe modelled as a vibrating physical object, computed directly to produce its sound rather than recording or sampling it.

Question it answers: What would this instrument sound like, computed from its physics rather than recorded?

Leaves out: Captures the physics of the instrument, not the decades of a player's technique.

Runs on: about 10^10 FLOPS rough runs in real time on a laptop or synthesiser Since: 1983 by 1983, digital waveguide models of vibrating strings

[Continuum fields](https://simulation-hypothesis.com/simulations/methods#m-continuum)[Differential equations](https://simulation-hypothesis.com/simulations/methods#m-equations)floor: Fields

### Generative art systems

Rules, randomness or simple agents run forward to make an image, pattern or composition. The rules alone didn't fully spell out the result in advance.

Question it answers: What shape does this process take if it is simply allowed to run?

Leaves out: The artist chooses which random runs to keep, so 'generative' work still involves substantial curation.

Runs on: about 10^9 FLOPS rough runs on a laptop in seconds to minutes Since: 1965 by 1965, early computer-generated art from algorithmic rules

[Rules and states](https://simulation-hypothesis.com/simulations/methods#m-rules)[Stochastic processes](https://simulation-hypothesis.com/simulations/methods#m-stochastic)floor: Rules

### Procedural and generative music

Musical rules, Markov chains or simple agents run forward to generate a melody, harmony or an evolving soundtrack that reacts to what is happening on screen.

Question it answers: What music does this set of rules produce if it keeps generating without a composer choosing every note?

Leaves out: Tends to sound competent and rarely surprising, which is exactly what the rules allow.

Runs on: about 10^9 FLOPS rough runs on a laptop in real time Since: 1957 the Illiac Suite, an early rule-based computer composition

[Stochastic processes](https://simulation-hypothesis.com/simulations/methods#m-stochastic)[Rules and states](https://simulation-hypothesis.com/simulations/methods#m-rules)floor: Rules

### Procedural texture and material generation

Surface patterns such as wood grain, rust, skin pores or rock strata generated from noise and simple physical rules, instead of painted by hand or photographed.

Question it answers: What should this surface look like close up, generated rather than photographed?

Leaves out: Noise-based textures can look uniform at a scale a real surface never would.

Runs on: about 10^10 FLOPS rough a workstation generating textures for one asset Since: 1985 by 1985, Perlin-style procedural noise textures in computer graphics

[Rules and states](https://simulation-hypothesis.com/simulations/methods#m-rules)[Stochastic processes](https://simulation-hypothesis.com/simulations/methods#m-stochastic)floor: Rules

### Physically based sound effects

Impact, rolling and scraping sounds synthesised from the shape, material and contact of the simulated objects instead of played from recordings.

Question it answers: Which sound does an object of this material make when it hits that surface?

Leaves out: Mode models are tuned by ear and omit radiation into a real room.

Runs on: about 10^9 FLOPS rough a laptop, real time Since: 2001 by 2001, van den Doel, Kry and Pai, FoleyAutomatic [source](https://doi.org/10.1145/383259.383322)

[Differential equations](https://simulation-hypothesis.com/simulations/methods#m-equations)[Real time](https://simulation-hypothesis.com/simulations/methods#m-real-time)floor: Fields

### Live-coded music and visuals

Performers write and change running code in front of an audience, so sound or image evolves as the program is edited.

Question it answers: How does music change when its rules are rewritten while playing?

Leaves out: The simulation is the program itself; nothing physical is modelled.

Runs on: about 10^8 FLOPS rough a laptop, real time Since: 2004 by 2004, the TOPLAP group formed to unite live coding communities [source](https://en.wikipedia.org/wiki/Live%5Fcoding)

[Rules and states](https://simulation-hypothesis.com/simulations/methods#m-rules)[Real time](https://simulation-hypothesis.com/simulations/methods#m-real-time)floor: Rules

### Neural scene rendering

A neural network fitted to a set of photographs stores a scene's light and density so that new views can be rendered from any camera position.

Question it answers: What would this place look like from where nobody took a photograph?

Leaves out: It interpolates between photographs and invents detail where none were taken, so views far from the originals are unreliable.

Runs on: about 10^15 FLOPS rough hours on a GPU for a scene, and a day for a large one. Since: 2020 Mildenhall and colleagues' NeRF, representing scenes as neural radiance fields for view synthesis. [source](https://doi.org/10.1007/978-3-030-58452-8%5F24)

[Learned emulators](https://simulation-hypothesis.com/simulations/methods#m-learned)[Differential equations](https://simulation-hypothesis.com/simulations/methods#m-equations)floor: Fields

[◂ Defence and conflict](https://simulation-hypothesis.com/simulations/defence-conflict)[Video games ▸](https://simulation-hypothesis.com/simulations/video-games)