How we used OKO to plan and execute the biggest cross-reality concert in Times Square

Bridging the gap between physical and digital spaces has always been a challenge. It requires technology that simplifies complexity rather than adding to it. Using insights from every mixed reality project we’ve worked on, we’ve built OKO, a suite of apps built on The Connected Spaces Platform (CSP). OKO breaks down the barriers between physical and digital spaces and enables creators to build things like digital twins. Typically, digital twins are pretty hard to create, tough to deliver, and even harder to anchor to their real-world counterparts. But this is something OKO does really well.

The OKO and CSP story began at the World Expo in Dubai, where we were challenged to share the experience of the physical site with a global audience in real-time. With no off-the-shelf tech available, we created a social, interactive digital twin of the 4.38km² site from the ground up to enable on-site and off-site visitors to experience the same digital layer in sync. We’ve since evolved that technology into a flexible open-source platform that can easily build and maintain digital twins, allowing teams to explore and plan accurate digital environments before anything is built, streamlining workflows like previs for production.

Exploring the role of digital twins in live event preproduction

This capability was put to the test in a groundbreaking project: planning and executing a cross-reality concert in the heart of Times Square, synchronized with a virtual gaming world. By using OKO’s digital twin technology, we efficiently coordinated a complex event in a way that would have been impossible using traditional methods.

Key considerations unique to the project dictated the use of a digital twin:

No live rehearsal was possible: Times Square’s dense foot traffic made live rehearsals logistically unfeasible. The concert’s secrecy also meant we couldn’t rehearse with the talent.
Independently controlled screens in Times Square: Each screen in Times Square had unique technical specifications, requiring meticulous planning to synchronize content across them all while preserving creative integrity. Exporting to multiple endpoints added further complexity – challenges best solved with a digital twin.
Camera and stage setup previsualization: The team needed to previsualize camera placements, movements, and stage layouts to optimize the live broadcast.

Screen synchronization and content management

Managing Times Square’s vast array of independently controlled screens (we used 48 on this project, the most ever utilized for one event) was a complex challenge. Visualizing the digital twin in OKO streamlined this process by:

Screen Identification

Every screen in Times Square has different labels e.g. the vendor’s name, the screen’s name, and its technical details. The OKO space incorporated a feature to display this data on the screens, allowing the team to make informed decisions about which screens to activate while balancing the needs for the creative vision. The team efficiently coordinated content delivery and playback by visualizing which screens belonged to which vendors.

Content Previews

The digital twin displayed previews of the content that had been UV-mapped to each screen, enabling the team to ensure creative consistency across all displays.

Camera Placement and Zoning

Using the digital twin allowed the team to visualize equipment placement in relation to screen content and broadcast camera movements in Times Square

Key highlights included:

Overhead Map Views
The OKO Web client’s overhead view feature allowed the team to suggest placements using a map underlay of Times Square. This enabled the sharing of 3D plans that helped identify key camera and equipment positions.
Interactive Control Panel
A client-facing control panel brought everything together, letting stakeholders interact with LED screen content, overhead maps, and camera views to plan and adjust broadcasts.

Stage Layout Setup

Previsualizing the stage setup was essential for both creative and logistical purposes. OKO supported this effort.

Stage Modeling

Using a Gaussian Splat model of the stage, the team replicated the stage’s dimensions as accurately as possible within the digital twin. This allowed the creative team to consider sizing and placement of stage elements like risers, lighting, and equipment.

Iterative Design

The platform enabled iterative testing of stage designs, ensuring a balance between creative vision and functional needs. By previsualizing stage components, the team could explore and refine configurations that best supported the experience.

Immersive Stakeholder Collaboration

To ensure alignment among stakeholders, OKO’s Web Desktop client was employed for:

Spatial Realism and Collaborative Reviews

The creative directors could immerse themselves in a realistic simulation of Times Square with the content mapped and placed spatially. The web-based experience made it easy for stakeholders in different locations to collaborate.

Informed Decision-Making

Viewing the setup allowed stakeholders to make informed decisions regarding screen content and selection, stage design, and camera placements.

How OKO improved the pre-production process

From content planning to on-site execution, OKO played a crucial role in streamlining pre-production. By enabling easy collaboration, real-time adjustments, and precise digital visualization, the platform helped teams work more efficiently and effectively, wherever they were.

Key takeaways:

Efficiency: By creating a detailed digital twin of Times Square, OKO streamlined coordination across creative and technical teams, allowing them to make quick, informed content decisions.
Adaptability: Real-time iterations enabled the team to adjust plans dynamically, ensuring that the creative vision was preserved while accounting for any impact on the event's physical layout.
Collaboration: The ability to collaborate effectively across distances – through OKO’s Web Desktop Client – resulted in a cohesive content vision and execution, regardless of team location.

Beyond Times Square

The success of the Times Square digital twin is just one example of how OKO can transform live event production. While this project presented unique challenges due to its high-profile location, the potential applications for OKO extend far beyond iconic spaces. Whether planning large-scale global events, designing complex installations, or managing production workflows remotely, creators can bring their visions to life more efficiently, effectively, and collaboratively through the use of a digital twin.

Smarter Live Event Production Through The Use Of Digital Twins