The future of visual storytelling: exploring virtual production
I sat down with AJ, our Director of Virtual Production to get the latest on the technology. We delved into the fundamentals of virtual production, how newcomers can dive into this evolving field, and the promising future it holds.
Virtual Production is a broad catch-all term – what areas do you and the team work on?
Magnopus has been fortunate enough to have been involved in virtual production for the past seven years. During that time, we’ve been involved in a variety of different projects, providing services across the entire spectrum.
For example, on The Lion King, we developed a Virtual Cinematography Stage that utilized existing film crews’ knowledge and expertise in their fields to operate physical film equipment which manifested in virtual cameras being moved in the same relation as their physical counterparts. The entire system was developed using the Unity real-time game engine and was used to shoot every single shot of the movie.
But typically, our initial approach to a project involves offering broad virtual production consultation to assist filmmakers in determining which aspects of this technology would best serve their needs. From there we help find stages, build out volumes (both LED and virtual cinematography), outsource specialized services or insource from our own VAD and Viz departments, operate those stages once in production, and assist with ancillary parts of the technology/cinematography components of production.
On the content side, we help create concept imagery for environments; we offer physical location scanning and capture services using both photogrammetry and lidar; 3D art development, generation, and design; previs shot development; techviz production planning, and postviz shot output. We also provide tool and software development for products, features, and systems that support all of these efforts, not only for our own use but also for the benefit of the entire industry.
Why is virtual production gaining traction?
Virtual production is generating a lot of buzz because it enables crews, filmmakers, and actors to immerse themselves in the digital worlds they’re shooting. In the past, such exposure and access were limited due to the extensive post-production processes of VFX. The next generation of virtual production use cases will focus on how the real-time assets created for film or TV can be leveraged across multiple distribution channels.
A virtual environment created for an exotic location can serve several purposes throughout the production process. Production designers can utilize it for set design, cinematographers can use previsualization for virtual blocking, stunt teams for stunt visualization, and camera departments and DPs for shot design, lighting, and equipment planning. Virtual production teams can use the asset for projection on the LED wall and potentially for final pixel in-camera VFX (ICVFX). However, a paradigm shift could soon occur in how the asset is utilized after production.
As the same asset is used by all these departments in a singular DCC (Unreal Engine) that the industry already uses to make immersive consumer-faced digital media, audience-centric interactive experiences become increasingly viable. Bringing audiences into these environments in a way that allows them to follow the story, explore the environment, and potentially even tell their own stories would grant them unprecedented access to these IPs.
More exciting still is the ability for audiences to influence these environments with their own actions. This opens up opportunities for second-screen experiences and the gamification of streaming, making it much easier to engage with audiences and encouraging them to participate actively.
We built a virtual environment for use in an LED volume while shooting parts of Westworld Season 4 Episodes 7 & 8. We worked closely with the Westworld Art Department to build a Times Square environment set 30 years in the future. Check out all the details here.
Where should someone start if they want to try virtual production?
Virtual scouting is the most straightforward entry point to virtual production. Traditional 2D monitor-based scouting can be challenging since humans are accustomed to perceiving spatial environments in three dimensions. When evaluating an environment via Zoom, PDF, or a screening room, people cannot engage all the cognitive processes needed to make fully informed decisions based on the surroundings.
By utilizing immersive AR camera tracking, such as Vcam, or VR headsets, virtual scouting can unlock the spatialization aspect of our brains, enabling us to fully understand the constraints and demands of the environments we’re shooting.
Blocking out a scene using virtual scene geometry as boundaries, understanding lighting conditions determined by the path of the sun, planning action scenes and how to stay on one side of the line are all immediately grasped by filmmakers and department heads upon immersing themselves into the virtual space.
As a result, decisions can be made much earlier in the production process, well before the crew sets foot on the physical set. Indecisiveness or delays in decision-making can be much costlier on the actual set. Virtual scouting serves as the initial entry point to the wider range of virtual production options, including virtual cinematography and immersive previsualization design, which we now refer to as "virtual blocking."
Once the filmmaker starts virtually exploring a 3D environment, it's natural to want to preview how the environment will appear on screen. Chess piecing some characters or vehicles through that environment is relatively straightforward, and with the ability to choose the lens, film back, and camera support mechanics, we can delve into the realm of virtual cinematography.
Are advances in VP relevant to all types of production – feature film, TV, TVC, music video etc?
Yes! Virtual production techniques allow filmmakers and content creators to create highly detailed and immersive digital environments that can be used in a variety of genres and formats.
For example, virtual production can be used to create realistic and detailed backgrounds for live-action scenes in feature films and TV shows or to create entire digital worlds for animated content. In TV commercials and music videos, virtual production techniques can be used to create visually stunning and highly stylized environments that may not be financially possible with traditional live-action production methods.
Virtual production can also be used to create pre-visualization (pre-vis) for complex action sequences in TV commercials and music videos, allowing filmmakers to plan and execute shots more efficiently and effectively. In feature film production, virtual production allows for the reproduction of digital environments live on set, whether for simulcam or ICVFX, giving DPs, Directors, and VFX Sups a vision of what the final pixels will be, allowing for a much more creative and iterative development process which happens far earlier in the production roadmap.
Virtual production can also be employed in real-time production, such as live events, particularly music concerts, where digital environments can be displayed on large screens to generate immersive experiences for audiences. Moreover, these assets can be personalized for each audience member, serving as an augmented reality (AR) asset that exists on the user's own device from their distinct perspective and vantage point.
How is the traditional film industry reacting to VP’s potential?
Like anything new, there are early adopters and there are those that will sit back and wait. But virtual production tools and techniques are rapidly evolving and demonstrating value so more filmmakers and studios will incorporate them into their workflows in the coming years. It’s rare to find a filmmaker or studio that is willing to dive headfirst into new technologies, whereas it’s more common to find a specific tool or system that solves a problem that was previously causing immense displeasure or frustration. As those filmmakers see the creative benefits of using a tool that works in real-time, the iterative advantages become evident, both creatively and financially.
What would you say are the biggest obstacles to adoption?
The investment in technology and training, the need for new workflows and techniques, and the risk of technological obsolescence are all potential barriers to someone adopting virtual production techniques. However, for those who are willing to make the initial investment and embrace new workflows, virtual production has the potential to revolutionize the filmmaking process and unlock new creative possibilities.
Looking ahead a few years, could virtual production eventually become the primary mode of production?
Nothing is more photoreal than photographing a real place with real people using real equipment. In this industry, we don’t always have that luxury, and there are many techniques to realize the creative vision of the filmmaker. Virtual production techniques are included in that larger box of tricks, some of which will be reserved only for the upper echelon of productions that can afford it, while others will become so commonplace that it’ll be hard to remember a time when they weren’t used.
Now the WGA strike has ended, do you think virtual production will be in demand to help get the TV and movie industries back on their feet?
Good question! I believe the demand for various aspects of virtual production will likely vary. For instance, LED wall production may not experience immediate high demand due to the time required for virtual asset creation and shoot planning. Productions will want to prioritize rapid filming to meet release content needs. However, areas like virtual cinematography, where iteration and visualization happen so fast, could see quicker adoption, with the potential to scale up within a matter of days or weeks.
