The Virtual Art Department (VAD) is increasingly becoming a standard in TV and filmmaking as it can provide great creative flexibility and efficiency over traditional methods. Here's a look at how a Virtual Art Department works and our experience of operating one for Fallout, the post-apocalyptic series produced by Kilter Films for Amazon Prime Video.

What is a Virtual Art Department?

A Virtual Art Department (VAD) is a team that uses digital technology to create virtual environments for film and television. Unlike traditional art departments that build physical sets and props, a VAD works with 3D modeling software and real-time game engines, like Unreal Engine, to design and visualize these elements in a virtual space.

The Importance of the VAD

By leveraging real-time technologies, VADs enable filmmakers to visualize and iterate on scenes live, making the creative process more flexible and efficient. This real-time feedback loop allows directors, cinematographers, and production designers to see and adjust digital environments and assets on the fly, ensuring a seamless blend with live-action footage. On the day of shooting, these elements are projected onto an LED wall where they can react to novel camera positions. Hence the phrase, in-camera visual effects (ICVFX).

Roles within our Virtual Art Department

A VAD brings together people with diverse skills to make the magic happen. On Fallout we used the following roles:

VAD supervisor: Oversees the artistic vision and ensures it aligns with the overall production design.
VAD Lead/Virtual Gaffer: Responsible for managing digital lighting setups and ensuring the virtual scenes are lit to achieve the desired artistic and realistic effects, while seamlessly matching on-set lighting.
VAD Lighting & Look Dev Supervisor: Oversees the overall visual aesthetics, including lighting and material properties, to ensure consistency and quality across all digital assets.
VAD Environment Lead: Manages the design, modeling, texturing, and rendering of virtual landscapes, sets, and backgrounds to ensure they align with the artistic vision and technical requirements.
VAD 2D Art Director: Responsible for the visual style and quality of all 2D artwork. This includes overseeing concept art, matte paintings, and other 2D elements.
VAD Matte Painter: Creates detailed and realistic digital backgrounds and environments that seamlessly integrate into virtual scenes. They paint 2D images or composite photographic elements to achieve the desired look, enhancing the visual depth and atmosphere of scenes.
VAD Artist: Creates and develops various digital assets, including 3D models, textures, and environments. Collaborates with other team members to ensure that digital assets align with the overall artistic vision and technical specifications, contributing to the overall visual storytelling of the production.
VAD Technical Artist: While both VAD Artists and VAD Technical Artists create digital assets, the latter focuses more on ensuring that these assets are efficiently created, integrated, and perform well within the virtual environment. Technical Artists optimize workflows, develop tools, troubleshoot issues, and ensure that digital assets are efficiently created and integrated without compromising performance or quality.
VAD Animation & Rigging: Responsible for creating character and object animations and developing the underlying skeletal structures (rigs) that enable those animations to be applied efficiently and realistically within the virtual environment.

How the VAD was used on Fallout

Script Breakdown and Environment Design

One of the first things we did was provide creative input for leveraging virtual production across the entire show. At the time, only the pilot had been written and our involvement on the project was much like a creative department head. We worked with the filmmakers to break down the scripts into scenes and environments that would benefit the most from in-camera visual effects and other virtual production techniques. They settled on key environments such as the picnic area and vault door scenes in Vault 33, the cafeteria in Vault 4, and the New California Republic’s base inside the Griffith Observatory. Additionally, any scenes involving the Vertibird were earmarked for LED process shots.

Virtual Set Construction

The VAD built all virtual sets entirely inside Unreal Engine. This approach allowed the filmmakers to use Unreal’s suite of virtual production tools throughout the creative process. Virtual scouting tools enabled the filmmakers to perform tech scouts in VR, block out action, and place cameras and characters in precise locations. This meticulous planning was crucial for creating a heatmap of the environment, helping to focus creative efforts on the most critical areas and optimize resources effectively.

Real-time Modifications and Flexibility

Working in real-time 3D was essential for the production. Unreal Engine offered the flexibility to make creative modifications to the set during pre-production and even on shooting days. For instance, during the shooting of the Vault Door set, the idea to dynamically change the lighting mid-shot was implemented swiftly by the VAD, showcasing the agility and responsiveness that real-time tools provided.

GIF from Fallout Official Trailer - Prime Video

In this pivotal scene, Lucy leaves the Vault for the first time. Authenticity for this moment is key – and finding the right blend between physical and virtual provides the audience with access to that authenticity. To that end, much of the set was physical – the handrails, the walkway, the plank, the control mechanisms, and even the door itself. Meanwhile, the interior walls of the vault were rendered virtually, extending the scope of the set. To make room for the giant moving vault door, we offset that set piece from the LED wall itself and filled the gap with floating LED panels instead. These were giant wild walls the crew could use to position at any point in the set to accommodate extreme angles. Mounting these panels with motion trackers allowed the team to dynamically update the image on screen no matter where they were placed – effectively creating a moving window into the virtual world. This streamlined production by eliminating the need for the crew to search for expansive caverns or construct large set pieces on a stage, all without sacrificing visual impact.

Seamless Integration with Physical Sets

A significant aspect of the VAD's work involved coordinating with the physical art department to ensure a seamless blend between physical and virtual elements. This included building 1:1 scale 3D versions of a number of sets in Unreal, even those that did not necessarily plan on using an LED volume stage. This work allowed the filmmakers to visualize scenes, compose shots, inform every department on the show about the creative intent, and help anticipate any potential difficulties. For scenes that did intend to use the LED volume stage, this close collaboration ensured that the digital set matched the physical set pieces, maintaining consistency in light and color, materials, and design.

We faced a unique challenge with this environment from episode 1. There is a cornfield inside of Vault 32 where most community activities take place, including Lucy’s wedding. The scene features layers of practical corn that extend into the virtual set, as well as apple trees, distant hallways leading to the neighboring vaults, and a synthetic nuclear-powered projection of a bucolic farm setting. The result is a sort of odd twist on LED volume production. The story calls for an in-world projection of virtual imagery onto a giant wall, meanwhile, as crew members on set, we are looking at that very same imagery projected onto an LED wall. At times, it really did feel as if we were inside the world of *Fallout*. This particular environment required multiple layers of virtual corn, six different lighting setups – as well as dynamic hooks to enable lighting cues in real-time – and a virtual projection surface capable of playing back 8k image sequences at 24fps.

On-set Workflow and Collaboration

On set, the VAD worked closely with multiple departments to ensure the highest fidelity for final imagery. While on-set Virtual Production Unreal Operators maintained camera tracking, adjusted frustums, loaded and unloaded scenes, ensured the proper functioning of the LED wall, and wrangled shot data, the VAD team could come in and adjust the look of the scene in real-time between shots. This allowed the VAD to operate much like a traditional on-set Art Department – responding to the ever-changing needs of production. This close collaboration was crucial in achieving the desired visual effects and maintaining the creative vision of the show.

Challenges

One of the biggest challenges faced by the VAD was shooting on 35mm film within an LED volume, a relatively unprecedented approach. This required extensive testing and fine-tuning of genlock, color calibration, and exposure settings. Despite these challenges, the VAD's expertise allowed for the successful capture of final pixels on set, minimizing the need for extensive post-production VFX work.

While there were challenges, the VAD we put together for Fallout worked because of meticulous planning and the seamless integration of virtual and physical elements. The ability to make real-time adjustments and collaborate closely with various departments ensured that the show maintained its unique visual style. The innovative use of Unreal Engine and other virtual production tools not only enhanced the storytelling but also set new benchmarks for virtual production workflows in the television industry.

How a Virtual Art Department (VAD) contributed to the Fallout TV show