Briefing Document: Ray Tracing and its Applications
This document provides a detailed overview of ray tracing, focusing on its application in computer-generated imagery (CGI) for TV shows, movies, and video games. It outlines the core principles of path tracing, computational challenges, and hardware solutions, drawing key information and quotes directly from the provided source, "raytracing.pdf".
1. Introduction to Ray Tracing and Rendering
Ray tracing is a fundamental computational process used in CGI and special effects to simulate how light interacts with and illuminates 3D models, transforming them into realistic environments. It is essential for creating the visual effects seen in modern TV shows and movies.
Key Facts:
- "Every new TV show and movie that uses computer-generated images and special effects relies on Ray Tracing." (00:31)
- Rendering simulates how light "bounce off of and illuminate each of the models, thus transforming a scene full of simple 3D models into a realistic environment." (00:31)
2. Path Tracing: The Industry Standard
Path tracing is the current industry-standard ray tracing algorithm for TV shows and movies. It is renowned for its realism but demands immense computational power.
Key Facts & Concepts:
- Computational Intensity: Path tracing requires "an unimaginable number of calculations." (00:57) For instance, rendering a single scene using the entire world's population performing one calculation per second would take "12 days of nonstop problem solving." (00:57)
- Historical Context: The algorithm was conceptualized in 1986, but it took "30 years before movies like Zootopia, Moana, Finding Dory and Coco could be rendered using path tracing." Even then, it required "a server farm of 1000s of computers and multiple months to complete." (01:23)
- Modeling 3D Scenes: Artists create scenes by modeling objects (islands, castles, dragons, etc.). These models, even with smooth curves, are broken down into "small triangles." (02:43) GPUs primarily work with 3D scenes made of triangles.
- Texturing: After modeling, artists "assigns a texture to it which defines both the color, as well as material attributes, such as whether the surface is rough, smooth, metallic, glass, water-like, or composed of a wide range of other materials." (03:10)
- Scene Setup: Models are positioned, and lights (like the sky and sun) are added and adjusted for intensity and direction to simulate time of day. A virtual camera is then added, and the scene is rendered. (03:44)
- Simulating Light: Path tracing "simulates how light interacts with and bounces off every surface in the scene, thereby producing realistic effects such as smooth shadows across the buildings or the way light interacts with the water and produces bright highlights." (03:44)
3. How Path Tracing Works: Rays from the Camera
Unlike real-world light (which emanates from a source), path tracing sends rays from a virtual camera into the scene, then traces their interactions.
Key Concepts:
- Camera-Centric Rays: "With path tracing we don’t send rays out from the sky or light source, but rather we send out rays from a virtual camera and into the scene." (04:55) This is because "only the light rays that reach the camera are useful." (04:55)
- View Plane and Pixels: The 2D image is represented by a "view plane" in front of the virtual camera, with the same pixel count as the final image (e.g., 8.3 million pixels for 4K). (05:20)
- Massively Parallel Operation: "Ray Tracing is a massively parallel operation because each pixel is independent from all other pixels." (06:16) This means calculations for different pixels can occur simultaneously.
- Primary Rays: For each pixel, a "thousand rays per pixel" (05:49) are sent from the virtual camera through a random point in the pixel and into the scene. These "primary rays" determine "what triangle and object do the rays first hit and what basic color should be in that specific pixel." (07:09)
- Illumination and Shading: The initial image from primary rays is "fairly flat colored." (07:35) The next step is to determine how the intersection point is illuminated by light sources, defining the pixel's brightness and shading.
- Direct Illumination: Light directly from light sources. (08:42)
- Indirect Illumination: Light that has bounced off other objects before reaching the point. (08:42)
- Global Illumination: The combination of direct and indirect illumination. (09:07)
- Shadow Rays for Direct Illumination: From the point where a primary ray hits an object, "shadow rays" are generated and sent towards each light source. "If there are no objects between the intersection point and a light source, then that means that this point... is directly illuminated by that light source." (09:43) Factors like brightness, size, color, distance, and surface direction are considered and multiplied by the object's RGB values to determine shading. (09:43)
- Secondary Rays for Indirect Illumination: To calculate indirect illumination, "secondary rays" bounce off the initial intersection point in various directions. These secondary rays then hit new surfaces, and from those new points, further shadow rays are sent to light sources. This process of bouncing secondary rays multiple times and sending shadow rays at each point helps "find different paths where light bounces off different surfaces and indirectly illuminates the original point where the primary ray hits." (12:27)
- Color Bleeding: "One additional benefit of indirect illumination and the use of secondary rays is that color can bounce from one object to another." (13:30)
- Material Properties: The direction secondary rays bounce depends on the object's material and texture properties (e.g., roughness, glass). (13:50) A "perfectly smooth surface with no roughness" becomes a mirror, while "a material has a roughness set to 100%" results in "entirely random directions." (14:01) Glass materials generate "additional refraction rays that pass through the glass." (14:44)
- Computational Scale: For animations, "20-minute animation requires over a quadrillion rays," (15:25) explaining why path tracing was considered computationally impossible for decades.
4. Addressing Computational Challenges
Two primary computational problems in ray tracing are the quadrillions of rays and identifying which triangle a ray hits first in a scene with millions of triangles.
Solutions:
- Bounding Volume Hierarchy (BVH):
- This technique efficiently solves the problem of finding which triangle a ray hits first. (17:16)
- Triangles are recursively divided into pairs of "bounding volumes" (boxes) until each small box contains only a few triangles (e.g., 6 triangles). (17:16)
- This forms a "binary tree or hierarchy." (18:53)
- Instead of checking against every triangle, a ray performs "simple ray-box intersection calculation" at each branch of the BVH. (18:53)
- Once the ray "finishes traveling through all the bounding volume branches, which is called BVH traversal, we end up with a small box of only 6 triangles." (19:10)
- The "ray-triangle intersection calculation" is then performed only on these few triangles. (19:29)
- BVHs "reduce tens of millions of calculations down to a handful of simple ray box intersections followed by 6 ray triangle intersections." (19:29)
- GPU Hardware Advancements:
- The second problem (handling billions of rays) is solved by "incredibly powerful GPUs." (19:51)
- Modern GPUs feature specialized "Ray Tracing or RT cores" alongside "CUDA or shading cores." (20:05)
- RT cores are "specially designed and optimized to execute Ray Tracing." (20:05)
- They contain a "BVH traversal section" that executes BVH traversal in nanoseconds and a "ray triangle intersection section" to quickly find the hit triangle. (20:25)
- RT cores "operate in parallel with one another and pipeline the operations so that a few billion rays can be handled every second." (20:43) This allows complex scenes to be rendered in minutes. (20:43)
- Comparison with Supercomputers: The NVidia 3090 GPU (2022, few thousand dollars) performs 36 trillion operations per second, surpassing the 12.3 trillion operations per second of the 2000-era ASCI White supercomputer (which cost $110 million). (21:11) This illustrates the "mind-boggling" amount of computing power now available in a consumer-grade graphics card. (21:51)
5. Ray Tracing in Video Games
While movies and TV use full path tracing, video games employ different methods to achieve real-time ray tracing due to performance demands.
Methods Discussed:
- Pre-computed Lighting with Low-Resolution Duplicates:
- A "very low-resolution duplicate of all the models in the scene is created." (25:34)
- Path tracing determines direct and indirect lighting for these low-resolution objects, and the results are "saved into a light map." (25:34)
- This light map is then "applied to the high-resolution version of the objects in the scene, creating realistic indirect lighting and shadows." (26:05)
- This method is used in Unreal Engine's Lumen renderer. (26:05)
- Screen Space Ray Tracing:
- This method "doesn’t use the scene’s geometries but rather uses the images and data generated from the video game graphics rendering pipeline." (26:35)
- It utilizes data like a "depth map" (distance of objects/pixels from the camera) and a "normal map" (direction each object/pixel is facing). (26:35)
- By combining the view screen, depth map, and normal map, an approximation of 3D object coordinates and pixel directions can be generated. (27:07)
- Ray tracing then bounces rays off pixels using this simplified screen-space 3D representation to calculate effects like reflections. (27:32)
- Limitations: A "problematic issue with screen space ray tracing is that it can only use the data that’s on the screen." (27:58) This means objects outside the camera's view or behind it cannot be reflected. (27:58)
- This type of ray tracing is used in games like Cyberpunk. (28:09)
6. Conclusion
Ray tracing, particularly path tracing, is a complex but essential technology for creating realistic CGI. Its evolution from a computationally impossible concept to a practical tool for desktop computers is a testament to advancements in algorithms like BVH and specialized GPU hardware. While full path tracing remains a domain for film and TV, adapted ray tracing techniques are increasingly integrated into video games to enhance visual fidelity.
