How nvidia graphics cards work

The Architecture and Application of Modern Graphics Processing Units

Executive Summary

The modern Graphics Processing Unit (GPU) represents a monumental leap in computational power, evolving from processing millions of calculations per second for early 3D games to tens of trillions for contemporary titles. This performance is achieved through a fundamentally different architecture than a Central Processing Unit (CPU). While a CPU is a flexible, high-speed processor excelling at sequential tasks (analogous to a jumbo jet), a GPU is a specialized, massively parallel processor designed to execute the same simple instruction across vast datasets simultaneously (analogous to a cargo ship). The physical architecture of a high-end GPU, such as the GA102 chip, features a hierarchical structure of over 10,000 specialized cores (CUDA, Tensor, and Ray Tracing) organized into clusters and multiprocessors. Manufacturing realities lead to a "binning" process, where chips with minor defects are sold as lower-tier products with some cores deactivated. This immense processing capability demands extremely high-bandwidth memory, leading to innovations like GDDR6X and HBM, which use advanced multi-level voltage signaling to achieve data transfer rates exceeding one terabyte per second. The GPU's computational model is built on the "embarrassingly parallel" principle, primarily using Single Instruction, Multiple Threads (SIMT) architecture to manage tasks like 3D vertex transformation. This parallel processing prowess has expanded the GPU's application far beyond gaming into fields like cryptocurrency mining (though now largely superseded by ASICs) and, most significantly, powering the matrix multiplication operations at the heart of artificial intelligence and neural networks.

--------------------------------------------------------------------------------

1. The Exponential Growth of Computational Demand

The performance requirements for real-time graphics rendering have grown by several orders of magnitude over the past few decades. This trajectory illustrates the immense computational challenge that modern GPUs are designed to solve.

• 1996 (Mario 64): Required approximately 100 million calculations per second.

• 2011 (Minecraft): Required approximately 100 billion calculations per second.

• Modern Titles (Cyberpunk 2077): Require a graphics card capable of around 36 trillion calculations per second.

To contextualize this scale, achieving 36 trillion calculations per second would require the combined effort of approximately 4,400 Earths full of people, with every individual completing one calculation each second.

2. Core Architectural Distinction: GPU vs. CPU

GPUs and CPUs are designed for different purposes, a difference best understood through the analogy of a cargo ship (GPU) versus a jumbo jet (CPU). The amount of data processed is the cargo capacity, and the rate of processing is the speed.

• GPU (Cargo Ship): Designed for high throughput. It processes a massive number of calculations at a relatively slower rate. It is highly specialized for simple, repetitive tasks (like basic arithmetic) and is less flexible. GPUs cannot run operating systems or directly interface with networks or input devices.

• CPU (Jumbo Jet): Designed for low latency. It performs a smaller number of calculations at a much faster rate. It is a flexible, general-purpose processor capable of running complex, varied programs and managing an entire computer system, including the operating system and network connections.

Feature	Graphics Processing Unit (GPU)	Central Processing Unit (CPU)
Primary Design	Massively parallel processing	Fast, sequential processing
Analogy	Massive cargo ship	Jumbo jet airplane
Core Count	Extremely high (e.g., >10,000 cores)	Low (e.g., 24 cores)
Task Flexibility	Low; specialized for simple instructions	High; runs a wide variety of programs
System Role	Specialized co-processor	General-purpose system brain
Ideal Workload	Large datasets requiring parallel execution (e.g., graphics, AI)	Varied tasks requiring rapid evaluation

3. Physical Architecture of the GA102 GPU

The NVIDIA GA102 chip, used in the 3080 and 3090 series cards, is a complex integrated circuit built from 28.3 billion transistors. Its architecture is organized hierarchically to manage thousands of processing cores.

Hierarchical Core Structure

The chip's computational resources are divided into progressively smaller units:

1. Graphics Processing Clusters (GPCs): The chip contains 7 GPCs.

2. Streaming Multiprocessors (SMs): Each GPC contains 12 SMs.

3. Specialized Cores: Each SM contains multiple core types, including:

    ◦ CUDA (or Shading) Cores: 32 per "warp" unit within an SM.

    ◦ Tensor Cores: 1 per "warp" unit within an SM.

    ◦ Ray Tracing (RT) Cores: 1 per SM.

A fully functional GA102 chip contains a total of 10,752 CUDA cores, 336 Tensor cores, and 84 Ray Tracing cores.

Core Functions and Design

Each core type is a specialized calculator for a specific function:

• CUDA Cores: These are simple binary calculators primarily used in video games. A single CUDA core contains approximately 410,000 transistors and is optimized for the Fused Multiply and Add (FMA) operation (A * B + C), completing one such operation per clock cycle. Half of the cores handle 32-bit floating-point numbers, while the other half handle either 32-bit integers or floating-point numbers.

• Tensor Cores: These are matrix multiplication and addition calculators. Their primary use is in geometric transformations for graphics and, more critically, for processing neural networks in AI applications.

• Ray Tracing Cores: These are the largest but least numerous cores, designed specifically to execute computationally intensive ray tracing algorithms for realistic lighting effects.

More complex operations like division or trigonometric functions are handled by a smaller number of Special Function Units (four per SM).

Manufacturing and Product Binning

The GA102 chip design is used across multiple graphics card models (3080, 3080 Ti, 3090, 3090 Ti), which have different prices and performance levels. This is a result of a manufacturing practice called "binning."

• During fabrication, microscopic defects (from dust or patterning errors) can render small parts of the chip non-functional.

• Due to the GPU's highly repetitive design, a defect in one SM does not affect the rest of the chip.

• Instead of discarding the entire chip, manufacturers test each one, find the defective regions, and permanently deactivate the nearby circuitry.

• Chips are then categorized or "binned" based on the number of fully functional cores.

Graphics Card Model	Functional CUDA Cores	Status
3090 Ti	10,752	Flawless GA102 chip
3090	10,496	Minor defects
3080 Ti	10,240	Moderate defects
3080	8,704	Equivalent of 16 deactivated SMs

In addition to core counts, these cards also differ in maximum clock speed and the quantity/generation of their graphics memory.

4. The Role of High-Bandwidth Graphics Memory

GPUs are "data-hungry machines" that require a continuous, high-speed flow of data to feed their thousands of cores. This necessitates specialized, high-bandwidth memory systems.

Data Throughput Requirements

When a game loads, 3D models and environmental data are moved from the system's solid-state drive into the graphics memory chips on the card. The GPU's small onboard cache (e.g., 6 MB L2 cache on the GA102) can only hold a tiny fraction of a scene, so data is constantly transferred between the GPU and the graphics memory. The 3090 graphics card features 24 GB of GDDR6X SD-RAM.

GDDR vs. System DRAM

Graphics memory is engineered for maximum bandwidth, in contrast to the DRAM that supports a CPU.

• Graphics Memory (GDDR6X): Utilizes a very wide bus. The 24 chips on the 3090 combine for a 384-bit bus width, achieving a total bandwidth of approximately 1.15 terabytes per second.

• System Memory (DRAM): Uses a narrower 64-bit bus width, resulting in a maximum bandwidth closer to 64 gigabytes per second.

Advanced Data Encoding (PAM4 vs. PAM3)

To push data transfer rates higher, modern graphics memory technologies transmit data using multiple voltage levels, going beyond simple binary 1s and 0s.

• PAM4 (Pulse Amplitude Modulation 4): Used in GDDR6X memory, this scheme employs four different voltage levels to send two bits of data simultaneously.

• PAM3 (Pulse Amplitude Modulation 3): Used in the next-generation GDDR7, this scheme uses three voltage levels to encode binary bits into ternary digits. This approach was chosen by industry engineers to reduce encoder complexity, improve the signal-to-noise ratio, and enhance power efficiency.

Specialized AI Memory (HBM)

For AI accelerators, Micron has developed High Bandwidth Memory (HBM), which prioritizes even greater bandwidth.

• Structure: HBM consists of stacks of DRAM chips connected vertically with Through-Silicon Vias (TSVs), forming a "cube of AI memory" that surrounds the AI processor.

• Performance (HBM3E): A single HBM3E cube can hold 24-36 GB of memory. A typical AI accelerator can be surrounded by up to 192 GB of this high-speed memory. Micron's HBM3E uses 30% less power than competing products. These AI systems can cost between $25,000 to $40,000.

5. The Computational Model: Parallel Processing at Scale

GPUs excel at solving problems classified as "embarrassingly parallel," where a large problem can be broken down into many smaller, independent tasks that require little to no coordination.

Practical Application: 3D Graphics Rendering (SIMD)

The core principle is Single Instruction, Multiple Data (SIMD). A single operation is applied to millions of data points in parallel. This is exemplified by vertex transformation in 3D graphics:

1. The Task: A 3D scene is composed of many objects (e.g., 5,629 objects), each built from thousands of vertices (e.g., 8.3 million total vertices). Each object's vertices exist in a local "model space." To render the scene, all vertices must be transformed into a common "world space."

2. The Instruction: The transformation for a single vertex involves a simple addition calculation.

3. The Parallelism: This single instruction is applied simultaneously to all vertices of an object, and this process is repeated for every object in the scene. A single frame may involve 25 million such addition calculations, all of which are independent and can be executed in parallel across the GPU's thousands of cores.

Execution Hierarchy and the Evolution to SIMT

The workload is managed through a structured hierarchy scheduled by the GPU's Gigathread Engine:

• Thread: A single instruction executed on a single CUDA core.

• Warp: A group of 32 threads.

• Thread Block: A group of warps handled by a single SM.

• Grid: A group of thread blocks computed across the entire GPU.

Modern GPUs have evolved from a strict SIMD model to a more flexible Single Instruction, Multiple Threads (SIMT) architecture.

• SIMD (pre-2016): All 32 threads in a warp executed in perfect lockstep.

• SIMT (current): The same set of instructions is sent to all threads in a warp, but individual threads can progress at different rates. Each thread is given its own program counter. This provides greater flexibility and efficiency, particularly when code contains conditional branches ("warp divergence"). Threads within an SM can also share data via a shared 128 KB L1 cache.

6. Applications Beyond Graphics Rendering

The massive parallel processing capabilities of GPUs have been leveraged for tasks far beyond video games.

Cryptocurrency Mining (Bitcoin)

Historically, GPUs were ideal for mining Bitcoin. The process involves repeatedly running the SHA-256 hashing algorithm with different input values (a "nonce") to find a random output number that begins with a specific number of zeros.

• Parallel Application: A GPU could run thousands of iterations of the SHA-256 algorithm in parallel, each with a different nonce. A card like the 3090 could generate approximately 95 million hashes (or "lottery tickets") per second.

• Obsolescence: This application is now dominated by ASICs (Application-Specific Integrated Circuits), custom-built machines that are vastly more efficient. A modern ASIC miner can perform 250 trillion hashes per second, equivalent to the power of 2,600 graphics cards.

Artificial Intelligence and Neural Networks

AI is a primary modern application for GPUs, directly utilizing the hardware's architecture.

• Core Operation: Neural networks and generative AI rely on trillions to quadrillions of matrix multiplication and addition operations.

• Tensor Core Function: The GPU's Tensor Cores are specifically designed for this task. They take three matrices, multiply the first two, and add the third, performing all calculations concurrently because all input data is available simultaneously. This makes GPUs the ideal hardware for accelerating AI workloads.