Matti Siekkinen and Teemu Kämäräinen describe a cloud rendering prototype in an arXiv paper that continuously converts a live Unity scene into a 3D Gaussian Splatting model. The server sends model updates over a socket, and the client renders its own camera view. Once scene data arrives, head and camera movements can appear without another server round trip, although scene changes and newly explored areas still depend on network updates.

Why streamed pixels lose the camera race

Cloud gaming and XR services usually render a server-selected camera view and encode the resulting 2D frames as video. Video codecs compress those frames efficiently, but each decoded image remains tied to an earlier camera pose. Clients can warp or reproject the latest frame to compensate for latency, with quality degrading during large movements, disocclusions, depth errors, and lighting changes.

3D Gaussian Splatting, or 3DGS, represents a scene as a collection of oriented 3D ellipsoids with position, scale, rotation, opacity, and color parameters. A GPU projects and blends these Gaussians to render new viewpoints in real time. Existing 3DGS pipelines commonly reconstruct a scene offline from many photographs; this prototype learns from a running game engine and updates the representation during play.

Unity supplies the geometry

The prototype divides work between a Unity application and a native C++ plugin: