China’s 4DV AI Just Turned 2D Video into 4D Magic-Lets You Control the Camera

Imagine watching a regular 2D video not just on a flat screen, but from inside the scene. You can walk around it, zoom into a conversation, change your viewpoint as if you’re filming it yourself and even hear the sound shift based on where you’re “standing.”

That’s exactly what China’s startup 4DV.ai just made possible with its latest release: a revolutionary tool that transforms everyday 2D video into a fully immersive 4D experience using a breakthrough technique called 4D Gaussian Splatting. And here’s the kicker, it runs right in your browser, optimized for modern devices.

What Is “4D Gaussian Splatting”?

Gaussian Splatting is a cutting-edge rendering method that reconstructs scenes using tiny, semi-transparent 3D blobs known as Gaussians. Each Gaussian contains color, depth, and opacity information. When combined, these blobs form richly detailed 3D scenes.

Now, take this 3D method and add time as a fourth dimension. That’s where 4D Gaussian Splatting comes in.

Unlike static 3D models, 4D Gaussian Splatting tracks how scenes change over time.

It builds a deformable space that evolves as the video progresses, allowing users to control the camera at any point during playback rewinding, panning, zooming, and rotating the viewpoint in real-time.

“It’s like walking around inside the memory of a video.”

The result is a living, breathing video experience fully explorable in space and time.

How It Works: Turning 2D into “Live” 4D

• Back-projection of pixels: Each frame in the 2D video is analyzed, and its pixels are mapped into a 3D environment using camera motion estimates and depth cues. Each mapped pixel becomes a Gaussian point.
• Temporal fusion: These Gaussian points are not just stored statically. They’re linked across frames using a time-aware deformation field. This means the entire video scene becomes a dynamic cloud of Gaussians that updates as time progresses.
• Camera freedom: Unlike traditional video where the viewpoint is fixed, users can now move the camera in any direction, much like a video game spectator mode. You can view a moment from multiple angles front, side, overhead, or behind.
• Spatial audio magic: Sound is also reconstructed to reflect your movement. As you change position in the virtual space, the audio perspective shifts making the experience deeply immersive.
• Browser-based performance: All of this runs inside a browser via a lightweight WebGL engine called Play Canvas. That means no downloads or headset just click and explore on compatible devices.

Why It Matters

For Film & Media

• Directors and editors can now shoot once and “edit” endlessly by changing the camera angle after the fact.
• Interactive movies could let viewers choose their own camera paths or switch perspectives between characters.

For Sports & Events

• Watch a goal from the striker’s POV, then rewind and see it from the goalkeeper’s angle.
• Replay key moments in history-concerts, matches, speeches as if you were there.

For Education

• Imagine a science teacher recording a lab experiment in 2D, then letting students walk around the beaker mid-reaction.
• Historical documentaries could reconstruct events in 4D, allowing learners to experience time and space together.

Background: From Gaussian Splatting to 4D Breakthrough

Until recently, Gaussian splatting was limited to static 3D scenes great for capturing places, not so much for dynamic video.

Then came the 4D-GS model, introduced in 2024 through research presented at CVPR. It included:

• Canonical scene modeling: One base scene is created, and motion is modeled via learned deformation.
• High FPS rendering: The system runs at over 82 frames per second at 800×800 resolution on powerful GPUs like the RTX 3090, producing dynamic 4D scenes with minimal lag.
• Massive compression: Smart Gaussian pruning and grouping techniques reduce data size while maintaining fidelity.

This makes real-time rendering of 4D video possible something that was a distant dream just a year ago.

Strengths & Limitations

Strengths	Limitations
Real-time in-browser experience	High-quality 2D input required
No VR gear or apps needed	Large datasets for long videos
Full camera and spatial audio control	Complex scenes may require preprocessing
Makes videos explorable and interactive	Still evolving—minor glitches may occur

The Future of 4D Video

The implications of 4DV.ai’s innovation go far beyond cool camera tricks.

Democratizing Immersive Content

By running in browsers, this tech brings immersive experiences to users with modern devices, lowering barriers to accessibility.

Game and Simulation Engines

Imagine combining 4D Gaussian Splatting with generative AI to simulate virtual environments walk through reconstructed scenes or explore AI-generated worlds.

Journalism and Storytelling

Reporters could record events and let audiences explore the scene from any angle, fostering deeper engagement.

Conclusion:

With 4D Gaussian Splatting, 4DV.ai isn’t just playing with pixels, it’s redefining how we experience video.

This tool turns everyday recordings into living worlds where space and time are both yours to explore. It blends cutting-edge research, intuitive design, and lightweight engineering into a product that pushes boundaries and is available now.

You can experience it with just a browser on a compatible device.