Virtual Production

What is virtual production?

Virtual production is an umbrella term for any workflow where computer-generated imagery is created, viewed or photographed during the shoot rather than after it. The version most people mean today is in-camera visual effects (ICVFX) on an LED volume, the setup popularized by The Mandalorian (2019) and Industrial Light & Magic's StageCraft stage.

An LED volume has four working parts:

• The wall. Hundreds of LED panels bolted into a curved wall, often with a ceiling of panels above, built inside a soundstage. The screens display the environment and also act as the main light source for the scene.
• The real-time engine. A game engine such as Unreal Engine renders the 3D environment at the frame rate of the camera. Environments can be full 3D worlds, photographed 2.5D plates, or a mix of both.
• Camera tracking. Markers or sensors report the camera's position, rotation and lens data many times per second. The engine redraws the part of the wall the lens sees, called the inner frustum, from the camera's exact point of view, so the background shifts with correct parallax as the camera moves.
• The brain bar. A team of engine operators, technicians and VFX artists at workstations beside the stage who move the sun, swap sets, relight the sky or nudge a mountain between takes.

Outside the frustum, the rest of the wall keeps showing the environment at lower detail. It is never seen sharply by the lens, but it lights the actors and throws reflections into eyes, visors, car paint and puddles, which is the part green screen could never give you.

Virtual production also covers workflows with no LED wall at all: previsualization in a game engine, virtual location scouting in VR, virtual cameras for animated and motion-capture films, and live composites on green screen. Together they shift a large share of the work traditionally done in post-production forward into pre-production, and it is now one of the core stages of the filmmaking process on large productions.

The main types of virtual production

Productions mix and match these, but it helps to name them separately because each changes different jobs on the crew.

LED volume (ICVFX)

The final background is photographed on the wall. What the camera records is, ideally, the finished shot, with only cleanup and seam work left for VFX. The Mandalorian is the reference example: large parts of its first season were shot in a volume, with DPs Greig Fraser and Baz Idoine lighting mostly with the wall itself.

Live compositing and simulcam

The actors stand in front of a green or blue screen, but the tracked camera feeds a real-time composite to the monitors, so the director and operator frame against the actual environment instead of a flat color. The key is still pulled in post; the benefit is that framing and eyelines are decided on set.

Virtual cameras and performance capture

For films built almost entirely in the computer, a director or cinematographer handles a tracked device that works like a camera inside the digital set. James Cameron's Avatar (2009) is the landmark here, and Jon Favreau's The Lion King (2019) staged and "photographed" its animated world with virtual cameras and VR scouting.

Previs, techvis and virtual scouting

The cheapest and most common kind. The team walks the digital set in VR, tries lenses, checks where the sun falls at 6 p.m., and exports camera positions and crane heights for the real shoot. It sits right next to the storyboard and the shot list in the planning process.

Why do filmmakers use LED volumes?

The idea is old: put the background behind the actors and photograph it with them. Studios did it with rear projection for decades of car interiors, and 2001: A Space Odyssey (1968) used large-scale front projection for its Dawn of Man landscapes. Oblivion (2013) projected sky footage on huge screens around its glass-walled Sky Tower set, so the actors and the reflective set were lit by the actual clouds. Gravity (2013) placed actors inside a box of LED panels that played the lighting of Earth and space around them, and First Man (2018) used a large LED screen outside the cockpit windows. LED volumes combine all of this with a camera-tracked, fully 3D background.

What it gives the filmmaker:

• Interactive light and reflections for free. A chrome helmet, a windshield or wet skin reflects the real environment, not a green spill that must be painted out. The glow of a sunset wraps around faces in the same color it has in the background.
• Actors see the world. Performers react to a horizon, a creature or a burning city instead of tape marks. Eyelines are right because there is something to look at.
• Control of time and weather. Golden hour lasts twelve hours. The same dusk can be repeated for every take, every coverage angle and every reshoot.
• Fewer company moves. A desert, a starship bridge and a snowy pass can be shot on the same stage in one week, which changes the math of a shooting schedule and travel budget.
• Decisions on the day. The director sees a near-final image on the monitor and edits can start from footage that looks like the film.

The cost is flexibility later. Whatever is on the wall is baked into the plate; changing the sky after the shoot is harder than on green screen. That is why productions that use volumes well tend to plan harder, not less.

How to shoot on an LED volume

The practical rules come from the physics of the screens and the lens.

Keep the wall out of focus. LED panels are a grid of tiny light points. Shot sharp, they produce moiré, a shimmering interference pattern with the camera sensor. Use a large-format sensor, a longer lens or a wider aperture so the wall sits in soft depth of field, and keep actors several meters in front of the screens. Finer pixel pitch (the distance between LEDs, around 2–3 mm on many feature stages) lets you get closer.

Build a real foreground. The seam where the wall meets the floor is the weak spot. Art departments dress the stage floor with sand, rocks, grass or deck plating that continues the digital set, and add physical props at mid-ground so the eye reads one continuous space. Well-made practical effects like smoke, dust and rain sell the blend.

Light with the wall, then add. The screens give soft, colored ambient light and top light from the ceiling. But LEDs built for display emit narrow red, green and blue spectra, which can render skin oddly. The gaffer usually adds full-spectrum fixtures for the key, hard sources for sun, and flags to stop the wall from flattening contrast. Match color temperature between the wall and the lamps before the actors arrive.

Respect the tracking. The inner frustum lags the camera by a few frames. Fast whip pans and quick handheld moves can reveal it. Moderate dolly, crane and slider moves show off parallax best. Lens data (focal length, focus, iris) must reach the engine so the virtual background defocuses and scales correctly.

Sync everything. Camera shutter, LED refresh and engine render all lock to the same timing signal (genlock). Without it you get banding or flicker, especially at high shutter angles or high frame rates.

Finish the world before shoot day. Environments must be built, lit and approved in advance, often in review sessions with the DP and production designer on the actual stage. Budget time for a pre-light day on the volume, just as you would on a cyclorama.

Virtual production vs. green screen

Both are forms of background replacement, but they sit at opposite ends of the schedule.

| | LED volume | Green screen | |---|---|---| | When the background is made | Before the shoot | After the shoot | | Light on actors | From the environment itself | From studio lamps, plus green spill to remove | | Reflections | Real, captured in camera | Must be faked or avoided | | What actors see | The world | A colored wall and tape marks | | Changing the background later | Hard, it's in the plate | Easy, it's a key | | Best for | Controlled dialogue scenes, vehicles, reflective surfaces, long magic hours | Big action, heavy destruction, wide shots with many digital extensions |

Many productions use both. A volume can show a green patch only inside the camera's frustum while the rest of the wall keeps lighting the set, giving a clean key with correct ambient light. And wide establishing shots are often completed in post even on shows built around a volume.

Planning a volume shoot with boards and AI

Because virtual production moves design decisions to the front, its paperwork is visual: a mood board for the look of each environment, a lookbook for the sky, color and time of day, storyboards that mark which shots use the wall, and previs frames for the brain bar and art department.

In FlashBoards you can keep all of it on one infinite board per sequence: reference frames from films shot on volumes, generated environment concepts for the wall, storyboard panels, and notes on lenses and pixel-pitch distance next to each setup. Generating a few AI versions of the same environment at different times of day lets the director and DP pick the light before anyone commissions the full 3D build, and the chosen frames go straight to the virtual art department as a brief. See the recipes below for a starting point, then compare the takes side by side on the board.

You can't generate a real LED volume, but you can generate the two things a volume shoot needs early: pictures of the setup for pitching and planning, and near-final frames that show how the in-camera shot should look. Put every take on one FlashBoards board next to your references and storyboard.

Common mistakes

• Models often draw a flat screen with a picture on it. Say "tall curved wall of LED screens" and "ceiling of LED panels" to get the wraparound volume.
• Without a stated floor set piece, the actor floats on a black stage floor. Name the sand, rocks or deck plating that bridges into the wall.
• In final-frame prompts the background comes out razor sharp and looks pasted in. Ask for soft focus and for reflections of the environment on faces, glass and metal.
• In video, if the camera doesn't move sideways the wall looks like a painted backdrop. Ask for a lateral dolly and describe near and far layers moving at different speeds.

Further reading

Noah Kadner, The Virtual Production Field Guide (Epic Games, 2019) · American Cinematographer, coverage of The Mandalorian (February 2020)