Film, media and art
Cloth, smoke, crowds and light, generated from physical models rather than animated by hand.
Before computers, film and animation approximated physics with wire rigs, miniatures and hand-drawn frames. Now simulation supplies the physics directly: draping cloth, moving hair, crowds that don't repeat, and light that travels the way it does in the real world. The methods of engineering and science are pointed at a story instead of a bridge or a reactor. They're tuned to look believable, not to be physically exact, and they often run offline, frame by frame, instead of in real time. Music and image share one idea: set up the underlying process and let the result emerge.
Cloth simulation
Fabric represented as a mesh of points and springs, draping, folding and tearing under gravity and movement.
Question it answers: How does this garment move on a moving body without an animator posing every fold?
Leaves out: Real fabric has a memory of its folds that most cloth solvers ignore.
Runs on: about 1012 FLOPS rough a render farm node for one shot's cloth passSince: 1987 by 1987, early mass-spring cloth models for computer animation
Hair and fur simulation
Tens of thousands of individual strands, each obeying its own bending and collision rules, moved by wind, gravity and the body beneath.
Question it answers: How does a full head of hair or a coat of fur move believably, strand by strand?
Leaves out: Strand-to-strand friction is approximated. Otherwise each simulation would take longer than a production schedule allows.
Runs on: about 1012 FLOPS rough a render farm node for one character's hair passSince: 1989 by 1989, early strand-based hair animation research
Fluid and smoke simulation for effects
Fire, smoke, water, dust and explosions computed as a fluid on a grid or in particles, styled for how they should look rather than for scientific accuracy.
Question it answers: How does this explosion, wave or smoke plume move and look on screen?
Leaves out: Directed to look good in frame, not to conserve energy or mass exactly.
Runs on: about 1013 FLOPS rough a render farm allocation for one effects-heavy shotSince: 1982 by 1982, early computer-generated fluid and particle effects for film
Destruction and debris simulation
Buildings, vehicles and objects fractured into pieces and thrown apart by rigid-body physics, tuned so the collapse looks dramatic rather than merely correct.
Question it answers: How does this structure break apart, and where do the pieces land?
Leaves out: Real materials fail unpredictably; these fracture patterns are pre-planned to look good.
Runs on: about 1012 FLOPS rough a render farm node for one destruction shotSince: 1995 by 1995, rigid-body fracture simulation used in film effects
Muscle and skin deformation
A digital body's skin and soft tissue sliding, bulging and wrinkling over an animated skeleton and muscles underneath.
Question it answers: Does the body of this creature or character move as though it had real soft tissue?
Leaves out: Skin sliding over bone is still mostly approximated, not properly simulated tissue.
Runs on: about 1012 FLOPS rough a render farm node for one character's body passSince: 1999 by 1999, muscle-based deformation rigs for computer-animated characters
Rigid body dynamics for animation
Solid objects that collide, stack, roll and settle under gravity and friction, used to animate props and secondary motion without hand-keying every bounce.
Question it answers: How does this pile of objects tumble and settle once it is disturbed?
Leaves out: Objects occasionally interpenetrate or jitter in ways no real thing would.
Runs on: about 1011 FLOPS rough a workstation for one shot's prop simulationSince: 1988 by 1988, rigid-body physics used for computer animation
Position-based dynamics
Objects such as cloth and soft bodies move by directly adjusting point positions to meet constraints, which is fast and stable.
Question it answers: How can deformable objects be simulated fast enough for interactive use?
Leaves out: Not physically exact and stiffness depends on step count.
Runs on: about 109 FLOPS rough Real-time on one CPU or GPUSince: 2007 Position based dynamics (2007) source
Ocean surface synthesis
A water surface generated from a spectrum of waves, used to render open sea with swell and foam.
Question it answers: What does a sea of given wind and depth look like?
Leaves out: It models surface shape, not the full flow below or the breaking of large waves.
Runs on: about 1012 FLOPS rough a workstation, minutes per frameSince: 1986 by 1986, Fournier and Reeves, A simple model of ocean waves source
Particle system effects
Fire, sparks, smoke and spray drawn as thousands of short-lived points, each with its own motion and appearance.
Question it answers: How can a fuzzy event be shown as a cloud of independently moving points?
Leaves out: Points do not push on each other, so true fluid behaviour is absent.
Runs on: about 1010 FLOPS rough a laptop, real timeSince: 1983 by 1983, Reeves, Particle systems (used in a wall of fire effect) source
Snow and granular material simulation
Snow, sand and similar materials simulated as points that carry stress, so they pile, break and clump like the real material.
Question it answers: How does snow pack, fracture and flow when pushed?
Leaves out: Grain-scale ice physics is replaced by an averaged material law.
Runs on: about 1014 FLOPS rough a render farm, hours per shotSince: 2013 by 2013, Stomakhin and others, A material point method for snow simulation source
Digital crowd simulation
Thousands of individual figures, each with simple movement and avoidance rules, filling a stadium, battlefield or street without being animated one by one.
Question it answers: How does a crowd this size move, flee or fight as a mass?
Leaves out: Every figure in the back rows tends to move a little too similarly.
Runs on: about 1012 FLOPS rough a render farm allocation for one crowd shotSince: 1997 by 1997, agent-based crowd systems used in feature film battle scenes
Digital humans and faces
A face's muscles, skin and micro-expressions modelled well enough to animate a synthetic performance that reads as a real person's.
Question it answers: Can this synthetic face convey emotion as convincingly as the real one it is based on?
Leaves out: The eyes and the corners of the mouth remain the most difficult regions to reproduce convincingly.
Runs on: about 1013 FLOPS rough a render farm allocation for one hero character's faceSince: 2001 by 2001, high-resolution facial capture and simulation for film
Procedural cities and landscapes
Streets, buildings, terrain and vegetation generated by rules and randomness from a small set of parameters, instead of being placed by hand.
Question it answers: How can an entire city or continent be populated with detail that cannot practicably be placed by hand?
Leaves out: Generated cities tend to be statistically plausible but lack any specific history.
Runs on: about 1012 FLOPS rough a workstation or render farm node for one environment buildSince: 1982 by 1982, fractal terrain generation used in computer graphics
Virtual production sets
A computer-generated environment displayed live on large screens behind actors and filmed in camera, with the scene reacting to how the lens moves.
Question it answers: Can the background react to the camera in real time so it does not have to be added afterwards?
Leaves out: The lighting match between the real actor and the screen behind them remains imperfect.
Runs on: about 1013 FLOPS rough a cluster of gaming GPUs driving the stage screens liveSince: 2019 by 2019, large LED-volume virtual production stages in wide use
Digital ecosystems and vegetation
Forests, grasslands and animal populations generated and animated together, with plants competing for light and animals moving through the result.
Question it answers: How does a whole simulated landscape of plants and creatures look and move together?
Leaves out: Simulated ecosystems are tuned to look lush, not to balance the way real ones do.
Runs on: about 1012 FLOPS rough a render farm node for one environment's ecosystem passSince: 2000 by 2000, procedural vegetation and ecosystem tools for film environments
Atmospheric and sky simulation
Clouds, fog, rain and sky colour generated from physical models of light scattering and moisture, for a scene's weather rather than the real thing.
Question it answers: What should the sky and weather in this shot look like, without shooting on the right day?
Leaves out: Tuned for how a sky should look, not for genuine atmospheric physics.
Runs on: about 1012 FLOPS rough a render farm node for one environment's sky passSince: 1986 by 1986, physically based sky and cloud rendering models
Neural avatars
A neural network trained on multi-camera captures of a person renders a lifelike, animatable face or body.
Question it answers: Can a virtual person be rendered indistinguishably from the real one?
Leaves out: Needs capture of each subject, and behaviour is not generated.
Runs on: about 1013 FLOPS rough Per-person trainingSince: 2018 Deep appearance models for face rendering (2018) source
Virtual idols and influencers
A fictional performer rendered in computer graphics and managed as a celebrity, appearing in media and social feeds.
Question it answers: Does an audience attach to a performer who has no body?
Leaves out: The persona is written by a team; no autonomous inner life is modelled.
Runs on: about 1011 FLOPS rough a workstation, rendering in near real timeSince: 1996 by 1996, Kyoko Date, a 3D CG idol debuted by Horipro source
Virtual concerts
A live performance staged in a shared virtual world or game engine, with avatars, stage effects and a remote audience.
Question it answers: Can a concert be shared by an audience that is only present as avatars?
Leaves out: Audience reaction and crowd presence are thin compared with a real venue.
Runs on: about 1012 FLOPS rough a data centre, real time for thousands of viewersSince: 2006 by 2006, Duran Duran performed in Second Life source
Digital de-ageing
A performer's face is rebuilt in 3D and retargeted to a younger or older age, driven by the actor's own acting.
Question it answers: Can a face be shown at another age while keeping the actor's expression?
Leaves out: The result is an appearance model; no ageing biology is simulated.
Runs on: about 1013 FLOPS rough a render farm, hours per shotSince: 2008 by 2008, The Curious Case of Benjamin Button, with Digital Domain effects source
Performance-captured digital characters
A computer-generated character animated by an actor's recorded body and facial movement, composited into live footage.
Question it answers: Can an actor's movement drive a creature that looks nothing like a person?
Leaves out: Muscle and thought are not modelled, only the captured motion.
Runs on: about 1011 FLOPS rough a workstation, hours per shotSince: 2002 by 2002, Gollum in The Lord of the Rings: The Two Towers source
L-system plant modelling
Plant shapes grown by repeatedly rewriting a string of symbols into branching structure, then drawn as trees and leaves.
Question it answers: What simple rewriting rules produce a given branching shape?
Leaves out: The rules fit shapes; the chemistry and light competition that drive real growth are absent.
Runs on: about 108 FLOPS rough a laptop, secondsSince: 1968 by 1968, Lindenmayer, Mathematical models for cellular interactions in development source
Machinima
Films shot by directing characters and cameras inside a real-time game engine instead of a physical camera or an offline renderer.
Question it answers: What can be filmed in a world that exists only in a game engine?
Leaves out: The engine decides what can be shown; acting is limited to what its controls allow.
Runs on: about 1011 FLOPS rough a gaming PC, real timeSince: 1996 by 1996, Diary of a Camper, a short made in the Quake engine source
Animatronic figures
Mechanical, motor-driven figures of people and animals that perform scripted movement and speech.
Question it answers: How convincingly can a machine act like a living creature?
Leaves out: Motion is replayed from a recording; there is no sensing or decision.
Runs on: no computer. a mechanical controller, no computationSince: 1963 by 1963, Disney's Audio-Animatronics, the Enchanted Tiki Birds source
Physics-based character control
A virtual body with joints and muscles is trained to move, jump or fight by reinforcement learning in a physics engine, copying motion-capture clips.
Question it answers: Can a character balance and move naturally, reacting to pushes, without hand-made animation?
Leaves out: Style comes only from the clips it copies, and rare motions and fine hand-work are poorly handled.
Runs on: about 1015 FLOPS rough days on a GPU workstation per skill.Since: 2018 Peng and colleagues' DeepMimic, example-guided deep reinforcement learning of physics-based character skills. source
Physically based light transport
Light rays traced backward from the camera, bouncing off and through every surface, to compute a physically consistent image.
Question it answers: How does light behave in this scene, bounce by bounce, rather than being approximated?
Leaves out: Converges slowly, so noise is still suppressed by heuristics rather than by taking more samples.
Runs on: about 1014 FLOPS rough a render farm allocation across many nodes for one frameSince: 1986 Kajiya's 1986 path tracing algorithm
Sound propagation and acoustics
Sound waves traced or modelled as they reflect, absorb and diffract through a virtual room, city street or battlefield, for film and game audio.
Question it answers: How should this space actually sound, given its shape and the materials in it?
Leaves out: Real ears and rooms have idiosyncrasies that acoustic models routinely idealise.
Runs on: about 1011 FLOPS rough a workstation for one scene's acoustic passSince: 1968 by 1968, ray-based room acoustics modelling
Physical modelling synthesis of instruments
A string, reed, drum skin or pipe modelled as a vibrating physical object, computed directly to produce its sound rather than recording or sampling it.
Question it answers: What would this instrument sound like, computed from its physics rather than recorded?
Leaves out: Captures the physics of the instrument, not the decades of a player's technique.
Runs on: about 1010 FLOPS rough runs in real time on a laptop or synthesiserSince: 1983 by 1983, digital waveguide models of vibrating strings
Generative art systems
Rules, randomness or simple agents run forward to make an image, pattern or composition. The rules alone didn't fully spell out the result in advance.
Question it answers: What shape does this process take if it is simply allowed to run?
Leaves out: The artist chooses which random runs to keep, so 'generative' work still involves substantial curation.
Runs on: about 109 FLOPS rough runs on a laptop in seconds to minutesSince: 1965 by 1965, early computer-generated art from algorithmic rules
Procedural and generative music
Musical rules, Markov chains or simple agents run forward to generate a melody, harmony or an evolving soundtrack that reacts to what is happening on screen.
Question it answers: What music does this set of rules produce if it keeps generating without a composer choosing every note?
Leaves out: Tends to sound competent and rarely surprising, which is exactly what the rules allow.
Runs on: about 109 FLOPS rough runs on a laptop in real timeSince: 1957 the Illiac Suite, an early rule-based computer composition
Procedural texture and material generation
Surface patterns such as wood grain, rust, skin pores or rock strata generated from noise and simple physical rules, instead of painted by hand or photographed.
Question it answers: What should this surface look like close up, generated rather than photographed?
Leaves out: Noise-based textures can look uniform at a scale a real surface never would.
Runs on: about 1010 FLOPS rough a workstation generating textures for one assetSince: 1985 by 1985, Perlin-style procedural noise textures in computer graphics
Physically based sound effects
Impact, rolling and scraping sounds synthesised from the shape, material and contact of the simulated objects instead of played from recordings.
Question it answers: Which sound does an object of this material make when it hits that surface?
Leaves out: Mode models are tuned by ear and omit radiation into a real room.
Runs on: about 109 FLOPS rough a laptop, real timeSince: 2001 by 2001, van den Doel, Kry and Pai, FoleyAutomatic source
Live-coded music and visuals
Performers write and change running code in front of an audience, so sound or image evolves as the program is edited.
Question it answers: How does music change when its rules are rewritten while playing?
Leaves out: The simulation is the program itself; nothing physical is modelled.
Runs on: about 108 FLOPS rough a laptop, real timeSince: 2004 by 2004, the TOPLAP group formed to unite live coding communities source
Neural scene rendering
A neural network fitted to a set of photographs stores a scene's light and density so that new views can be rendered from any camera position.
Question it answers: What would this place look like from where nobody took a photograph?
Leaves out: It interpolates between photographs and invents detail where none were taken, so views far from the originals are unreliable.
Runs on: about 1015 FLOPS rough hours on a GPU for a scene, and a day for a large one.Since: 2020 Mildenhall and colleagues' NeRF, representing scenes as neural radiance fields for view synthesis. source