Beyond the Point Cloud: Union VFX and the Evolution of Gaussian Splatting in Feature Film
In the fast-evolving landscape of visual effects, few technologies have generated as much industry fervor as Gaussian splatting. While initially regarded as a novel visualization technique for static environments, a groundbreaking collaboration between Union VFX and Clear Angle Studios is transforming this mathematical model into a robust, production-ready solution for high-fidelity crowd simulation and digital human performance.
In the latest episode of the fxpodcast, VFX and Technical Supervisor David Schneider detailed how Union VFX is bypassing the traditional, labor-intensive hurdles of digital character creation by leveraging 4D Gaussian splatting. By marrying machine-learning-driven capture with a rigorous, art-directable pipeline, the team is successfully integrating real human performances into massive, relightable, and photorealistic crowd sequences.
The Core Innovation: Moving Beyond CG Crowds
For decades, the standard for creating large-scale digital crowds has remained largely unchanged: model, rig, animate, and deploy through complex agent systems. While effective, this pipeline is notoriously resource-heavy, often requiring significant time for animation cleanup to avoid the “uncanny valley” effect. Furthermore, traditional "card-based" crowds—using flat 2D footage of actors—frequently fail when the camera moves or when the scene’s lighting conditions conflict with the original capture.
Union VFX’s approach, developed in partnership with Clear Angle Studios, pivots away from purely synthetic assets. Instead, they are capturing actual human performances within a specialized "Volumetric Capture Rig" (VCR). These captures are then processed into 4D Gaussian splats—a method that represents 3D space as a collection of anisotropic points (splats) that maintain spatial and temporal relationships.
Unlike raw Gaussian splats, which are often prone to "jitter" and lack temporal stability, Union’s workflow treats the splats as a hybrid asset. By applying sophisticated image segmentation and denoising algorithms, the studio can isolate individual body parts—hair, skin, jackets, and trousers—turning them into Cryptomattes. This allows compositors to manipulate the appearance of crowd members with the same granularity as traditional CG elements, enabling color variation and seamless integration into any lighting environment.
Chronology: From FMX Concept to Production Reality
The development of this workflow was not an overnight success. It began with a directive from Union VFX co-founder and executive supervisor Adam Gascoigne, who identified a need for more efficient, realistic crowd solutions during the pre-production phase of a major upcoming feature.
- The Investigative Phase: Over the past year, the Union team conducted deep research into the viability of Gaussian splatting, specifically seeking to solve the "relighting problem."
- The Clear Angle Partnership: Recognizing the need for high-quality, controlled data, Union teamed up with Clear Angle Studios. Clear Angle, which had been developing its own R&D pipeline in conjunction with NVIDIA, constructed the "VCR"—a half-dome rig equipped with 40 synchronized machine-learning cameras and integrated, adjustable LED lighting.
- The Proof of Concept: The team conducted several stress tests, including the "cyclist on rollers" experiment. By capturing a cyclist on static equipment and then transforming that data through a virtual environment, they proved that performances captured in a neutral, stationary volume could be re-animated and placed into a scene with the correct physics and speed.
- Public Unveiling: Following successful internal testing and a presentation at FMX in Germany, the studio hosted an industry event in London, where they demonstrated the capabilities of the VCR to fellow VFX supervisors and producers.
Supporting Data: Why Splats are Production-Friendly
One of the most persistent criticisms of emerging AI-based visual technologies is their lack of performance efficiency. However, the data coming out of Union’s pipeline suggests the opposite.
Rendering Efficiency
Because Gaussian splats function as primitive shapes rather than high-polygon meshes, the memory overhead is remarkably low. During their testing, the Union team utilized Karma XPU—a renderer capable of balancing GPU and CPU resources. They reported that a 2K quality frame could be rendered in approximately 20 minutes on a GPU-enabled machine, a figure that is highly competitive with traditional CG crowd solutions.
The Power of "Instances of Instances"
The scalability of this approach is achieved through a hierarchical instancing system. A single captured performance can be broken down into primitive building blocks; a crowd scene then acts as an instance of these blocks. This allows for thousands of individuals to populate a stadium or plaza without requiring exponential increases in render time or memory usage.
Proximity and Detail
Traditionally, crowd assets are relegated to the background, hidden behind atmosphere, smoke, or foreground extras. Union’s testing has shown that these splat-based characters can hold up at much closer proximity—roughly three-quarters of the screen height—without breaking the illusion of realism.
Official Responses and Methodology
During the fxpodcast conversation, David Schneider emphasized that the strength of this approach lies in its "deterministic" nature. Unlike generative AI, which can be unpredictable and difficult to control, the Union/Clear Angle pipeline provides supervisors with total control over the output.
"With generative AI, you don’t always have a great understanding of what is being generated," Schneider explained. "Our process is deterministic. We know exactly what the outcome will be. The director and VFX supervisor can make specific decisions about where people go, how they behave, how they are lit, and how they are varied. In production terms, that control is everything."
The workflow also includes a Nuke-based editorial bridge. Clear Angle delivers "sprite-like" views of the captured data, which are then integrated into a USD-based pipeline. This allows editorial and layout departments to make creative decisions using the final-quality assets from the start of the process, rather than relying on placeholder proxies.
Implications: The Future of Digital Humans
The success of this workflow has significant implications for the future of film production. While the current focus is on crowd work, the potential applications extend far beyond stadium extras.
Beyond the Crowd
Schneider points to stunt work and superhero films as the next frontier. The ability to capture an actor on wires or performing complex movements, then process that performance into a relightable 4D splat, could render the need for traditional "digi-doubles" obsolete for many sequences. This would provide a level of photographic realism that is currently difficult to achieve with purely hand-animated models.
A New Standard for Virtual Production
By solving the relighting issue—allowing splats to cast and receive shadows within a CG environment—Union has essentially bridged the gap between raw photographic capture and digital manipulation. This signals a shift toward a future where "performance capture" is not just about the motion, but about the volumetric essence of the actor.
As the industry moves away from the hit-or-miss nature of early AI-generated media, the focus is clearly shifting toward systems that offer the "photographic truth" of Gaussian splatting with the "production control" of traditional VFX. If Union VFX’s results are any indication, the next generation of digital humans will be built not from polygons, but from the captured, relightable light of real-world performances.
