Research on Generative AI-based Reconstruction of Cinematic Images Guided by 3D Spatial Information in Unreal Engine

With the advancement of generative AI video technology, its application in film and television visual effects production has become increasingly prevalent. However, in complex 3D scenes, generated outputs often struggle to accurately reproduce spatial structures while maintaining logical consistency. This study employs high-altitude building shots from the trailer of Spider-Man: Into the Spider-Verse as a case study to investigate the impact of Unreal Engine (UE) 3D spatial guidance on the reconstruction of cinematic shots using generative AI. First, we analyze the spatial relationships among buildings, characters, and cameras in the original footage and then construct corresponding 3D scenes within UE. Subsequently, AI-generated images are produced using two approaches—with and without UE assistance—and compared with the original footage across multiple dimensions, including spatial relationships, composition, perspective, and camera angles. This research aims to explore whether UE 3D spatial guidance can enhance the ability of generative AI to reconstruct original cinematic shots. Preliminary results indicate that UE spatial guidance significantly improves the consistency of generated outputs in terms of composition, spatial relationships, and camera angles, while also improving perspective to a limited extent.

With the development of generative AI, its applications in video production have become increasingly widespread. However, existing text-to-video and text-to-image-to-video methods still rely primarily on textual prompts to control spatial relationships within shots, making it difficult to accurately convey camera positions, angles, and scene depth. Existing spatially controllable generation methods, such as CameraCtrl[1], rely on numerical camera pose inputs; CineMaster[2] employs 3D bounding boxes as object proxies; and LooseControlVideo[3] demonstrates the effectiveness of sparse directed 3D boxes. However, none of these methods directly use 3D scenes constructed in game engines as spatial guidance signals for AI video generation. As a real-time 3D creation tool, Unreal Engine enables the rapid construction of spatial relationships within scenes and the precise export of camera parameters, providing a practical medium for spatial guidance. Based on this, this study utilizes Unreal Engine (UE) to construct a 3D spatial scene and employs it as a spatial reference for AI image generation to reconstruct a target frame from the film Spider-Man: Brand New Day (2026). This study compares AI-generated results produced directly with those generated under UE spatial guidance, and conducts comparative analyses with the original cinematic frame across multiple dimensions, including spatial relationships, composition, perspective, and camera angles.

This study uses high-altitude building shots from the trailer of Spider-Man: Brand New Day as a case study to establish an AI image generation workflow guided by 3D spatial information in Unreal Engine (UE), as illustrated in Fig. 1. The overall workflow consists of three stages: target shot analysis, UE scene construction and spatial condition extraction, and AI generation.

First, the original shot is analyzed in terms of its spatial and cinematographic parameters. Four types of information are extracted: the composition and spatial positions of the character and buildings within the frame, camera angle, camera perspective, and lens focal length and field of view. These four types of information serve both as the basis for constructing the UE scene and as evaluation dimensions for the final generated results.

A corresponding 3D scene is constructed in Unreal Engine based on the above analysis, as shown in Fig. 2. The character and buildings are represented using simplified geometric primitives, with particular emphasis on reproducing their spatial relationships, including the character's position, the relative distance and depth between building volumes, and the position and orientation of the camera. Camera parameters are configured using UE's Cine Camera Actor, including camera position, rotation, and focal length, to align the perspective with that of the original shot. After the scene is constructed, an image is rendered from the camera's viewpoint and used as a spatial reference for AI image generation.

Two generation paths are established for the experiment. Path A provides both the UE scene screenshot and the text prompt as inputs to ChatGPT, while Path B uses only the text prompt without any spatial reference. The two paths are conducted using two separate accounts with the same generation model and identical text prompts, ensuring that the difference between the two conditions is attributable solely to the UE spatial guidance. The generated results are then compared with the original cinematic frame across four dimensions: spatial relationships, composition, perspective, and camera parameters.

Four groups of high-altitude building shots from the trailer of Spider-Man: Brand New Day were selected as case studies. Each shot followed the procedure described in Section 2: the spatial and cinematographic parameters of the original shot were first analyzed, followed by the construction of a corresponding 3D scene in UE and the capture of a scene screenshot. Images were then generated separately using the UE-guided and AI-only approaches. Both approaches used the same generation model and identical text prompts to ensure that the difference was attributable solely to the UE spatial guidance. The generated results were compared with the original cinematic frames across four dimensions: spatial relationships, composition, perspective, and camera angle.

The comparison results for the four shot groups are presented in Fig. 3. Overall, the UE-guided approach outperformed the AI-only approach in terms of spatial relationships and composition, while improvements in perspective and camera angle varied across shots. Taking Picture 1 as an example, the AI-only approach produced clear errors in the spatial relationship between the character and the buildings, as well as substantial deviations in composition. After UE guidance was introduced, the spatial positions of the character and buildings, the compositional proportions, the perspective direction, and the camera height were largely consistent with the original frame. This indicates that the UE scene screenshot provided an explicit spatial reference, enabling the model to better preserve the relative spatial relationships between the subject and its environment during generation. Pictures 3 and 4 showed similar trends. After UE guidance was applied, the results approached the original frame across all four dimensions; therefore, these two cases are not discussed further. Picture 2 presented a relatively special case. This shot features an extreme top-down perspective, complex interaction between the character and buildings, and dense spatial layering. After UE guidance, the spatial relationships and composition were improved, but the perspective and camera angle still deviated from the original. One possible explanation is that the shot contains complex depth relationships, while the geometric representation of the UE scene was limited. As a result, the AI model may have had difficulty accurately inferring the perspective of building surfaces and the camera's pitch angle from the screenshot. In addition, the near-vertical viewing angle makes the vertical spatial relationships within the image more complex, further increasing the difficulty of spatial interpretation for the generative model. The experimental results indicate that UE 3D spatial guidance produced the most significant improvements in spatial relationships and composition, with three of the four shots reaching or approaching the level of the original frames after guidance. For perspective and camera angle, the guidance was effective for shots with relatively conventional viewpoints but showed limited improvement for the extreme-viewpoint shot. These results suggest that the effectiveness of UE spatial guidance is influenced by both the accuracy of scene construction and the complexity of the camera viewpoint. When the constructed scene sufficiently represents the spatial structure and the viewpoint falls within a range that the model can effectively interpret, spatial guidance can improve reconstruction consistency; otherwise, its effectiveness may be reduced.

This study investigated the impact of Unreal Engine (UE) 3D spatial guidance on the reconstruction of cinematic images using generative AI, using four high-altitude building shots from the trailer of Spider-Man: Brand New Day as case studies. The experimental results indicate that UE spatial guidance significantly improves the consistency between the generated results and the original footage in terms of spatial relationships and composition. In terms of perspective and camera angle, UE guidance is effective for shots with conventional viewpoints, but provides limited improvement for shots with extreme viewpoints. These results demonstrate that using a 3D scene constructed in UE as spatial guidance can provide generative AI with more explicit spatial constraints than text prompts alone. However, its effectiveness is jointly constrained by the accuracy of scene construction and the complexity of the camera viewpoint. Future work can further explore higher-precision scene representations and spatial guidance strategies specifically designed for extreme viewpoints.