NVIDIA H100 vs H200 vs A100 vs B200: Architecture, Benchmarks, and Cloud Rental Economics

The Generational Silicon War in the Cloud

In the high-stakes arena of artificial intelligence infrastructure, compute hardware is not a fungible commodity. A common error among engineering leaders is treating GPU rental as a simple comparison of gigabytes of VRAM. Choosing between an NVIDIA Ampere A100 (80GB), a Hopper H100 (80GB), a Hopper H200 (141GB), or the next-generation Blackwell B200 (192GB) fundamentally dictates your model throughput, training convergence speed, and infrastructure burn rate.

Modern Large Language Models (LLMs) such as Llama 3.1, DeepSeek V3, Mistral Large, and Qwen 2.5 operate in two fundamentally different hardware regimes: compute-bound prefill (ingesting the prompt) and memory-bandwidth-bound decode (generating tokens autoregressively). Understanding how tensor core architecture, high-bandwidth memory (HBM) generations, and precision formats (FP16, FP8, FP4) interact with these regimes is essential for building a cost-effective cloud strategy.

This comprehensive technical guide dissects the architectural breakthroughs across NVIDIA’s Ampere, Hopper, and Blackwell microarchitectures, presents verified real-world benchmarks, and models the true cloud rental economics across leading providers including Lambda, Runpod, CoreWeave, and OCI.

Architectural Breakdown: GA100 vs. GH100 vs. GB200

To understand why performance diverges so dramatically across generations, we must examine the underlying silicon innovations and physical packaging:

1. Ampere GA100 (A100): The Foundation of Modern Deep Learning

Launched on TSMC’s 7nm process, the GA100 established the architectural standard for AI datacenters. It introduced 3rd-generation Tensor Cores supporting TensorFloat-32 (TF32), Bfloat16, and FP16, alongside 3rd-generation NVLink offering 600 GB/s bidirectional bandwidth. However, the A100 lacks hardware support for 8-bit floating-point (FP8) computation and caps out at HBM2e memory bandwidth (up to 2.0 TB/s on the 80GB SXM4 variant). Its streaming multiprocessors (SMs) rely on traditional asynchronous copy instructions that require manual shared memory staging.

2. Hopper GH100 (H100 & H200): The Transformer Engine Revolution

Manufactured on a customized TSMC 4N process with 80 billion transistors, Hopper represents the single largest generational leap in deep learning history. Hopper introduced four transformative technologies:

• 4th-Gen Tensor Cores with Transformer Engine: Dynamically adjusts precision between FP8 (E4M3 for forward pass, E5M2 for backward gradients) and FP16 on a per-layer basis during training and inference, doubling arithmetic throughput without sacrificing mathematical convergence.
• Tensor Memory Accelerator (TMA): A dedicated hardware unit on each SM that transfers multidimensional tensors directly between global HBM and shared memory (SMem) asynchronously, freeing CUDA warp execution threads from address calculation and indexing loops.
• DPX Instructions for Dynamic Programming: Accelerates algorithms like Smith-Waterman by up to 7x for biological sequence alignment and routing optimization.
• 4th-Gen NVLink & NVLink Network: Delivers 900 GB/s of all-to-all GPU-to-GPU bandwidth per card, expanding multi-node clustering capabilities.

While the H100 utilizes 80GB of HBM3 at 3.35 TB/s, the updated H200 swaps in 141GB of cutting-edge HBM3e memory, boosting bandwidth to an astonishing 4.8 TB/s. This 43% bandwidth increase directly resolves memory bottlenecks in LLM decoding.

3. Blackwell GB200 / B200: Hyperscale Extreme Density

NVIDIA’s Blackwell architecture combines two reticle-limited dies interconnected by a 10 TB/s two-way coherent chip-to-chip link (NV-HBI), packing 208 billion transistors. Blackwell introduces 2nd-generation Transformer Engines supporting micro-tensor scaling and native FP4 precision. With 192GB of HBM3e at 8.0 TB/s and 5th-Gen NVLink operating at 1.8 TB/s, a single B200 GPU can deliver up to 20 PFLOPS of FP4 compute.

The flagship rack implementation, the GB200 NVL72, liquid-cools 72 Blackwell GPUs and 36 Grace CPUs into a unified exascale supercomputing cluster acting as a single massive virtual GPU with 13.5 TB of aggregate HBM3e memory and 130 TB/s of bisection bandwidth.

The Silicon Packaging: CoWoS and the Memory Bandwidth Wall

To truly understand GPU cloud supply and performance differences, we must inspect TSMC's CoWoS (Chip-on-Wafer-on-Substrate) packaging. A modern AI accelerator is not a monolithic silicon chip mounted directly onto a motherboard. The high pin counts required by High-Bandwidth Memory (thousands of parallel microscopic traces) cannot fit onto conventional organic PCB substrates.

Instead, TSMC fabricates a passive silicon interposer—an ultra-thin layer of silicon with microscopic copper interconnects. The main compute die (or twin dies in Blackwell) is placed onto the interposer, surrounded by 6 to 8 stacks of High-Bandwidth Memory (HBM3 or HBM3e). Micro-bumps measuring barely 25 to 55 microns connect the logic die to the memory stacks.

Because these memory traces are millimeters long rather than centimeters long, they operate at lower electrical voltages, produce far less heat per bit transferred, and achieve bus widths exceeding 5,120 bits. This is why an H100 achieves 3.35 TB/s and an H200 achieves 4.8 TB/s, whereas discrete GDDR6X on consumer cards tops out at 1.0 TB/s.

The Roofline Model: Arithmetic Intensity & Execution Bottlenecks

To determine which GPU will execute your specific model faster and cheaper, you must apply the Roofline Model. The Roofline Model plots computational performance (TFLOPS) against Arithmetic Intensity (measured in FLOPs performed per byte of memory read from HBM):

Arithmetic Intensity (FLOPs/byte) = Total Computation (FLOPs) / Total Memory Transferred (Bytes)

- If Arithmetic Intensity < Ridge Point: The workload is MEMORY-BANDWIDTH BOUND.
  Adding more TFLOPS does nothing; performance scales strictly with memory bandwidth (HBM3/HBM3e).
- If Arithmetic Intensity > Ridge Point: The workload is COMPUTE BOUND.
  Performance scales directly with Tensor Core clock rates and FP8/FP16 arithmetic execution units.

How LLMs Traverse the Roofline

• Prompt Ingestion (Prefill Phase): High Arithmetic Intensity. The model processes hundreds or thousands of input tokens simultaneously in a single dense matrix multiplication (GEMM). This phase is heavily compute-bound. Hopper’s 4th-Gen Tensor Cores and FP8 Transformer Engine cut prefill latency by up to 3.5x over Ampere.
• Token Generation (Decode Phase): Extremely low Arithmetic Intensity. The GPU must read all 70 billion parameters (140GB in FP16 or 70GB in FP8) from HBM into the compute registers just to generate a single token for a single user! At batch size 1, the GPU spends 95% of its time waiting for memory reads. Here, the 4.8 TB/s bandwidth of H200 delivers 45% faster generation than H100, even though both chips possess identical compute core counts.

Architectural & Technical Specifications Matrix

Deep Dive: The FP8 Precision Revolution (E4M3 vs. E5M2)

Why is FP8 on Hopper so much faster than FP16 on Ampere? The Transformer Engine does not use a single naive 8-bit format. It dynamically juggles two distinct IEEE floating-point representations:

• E4M3 (1 sign bit, 4 exponent bits, 3 mantissa bits): Maximizes numerical precision at the cost of dynamic range. Used exclusively during the forward pass of transformer layers and for inference activations. Because activations are bounded, the higher precision preserves fine nuances in attention distributions.
• E5M2 (1 sign bit, 5 exponent bits, 2 mantissa bits): Matches the dynamic exponent range of standard FP16 (5 bits of exponent), but has lower mantissa precision. Used during the backward pass for calculating gradients, which can vary wildly across multiple orders of magnitude.

The Hopper hardware monitors tensor maximum values on every layer and calculates an automatic scaling factor ($\text{scale} = \text{max\_val} / \text{fp8\_max}$). This ensures that tensors are multiplied at 8-bit rates inside tensor cores while accumulator registers accumulate in full 32-bit floating point, guaranteeing mathematical fidelity.

FlashAttention-3: Unlocking Hopper’s Full Potential

While FlashAttention-2 optimized IO awareness on Ampere, FlashAttention-3 was engineered from scratch specifically for the Hopper architecture. It introduces three low-level hardware optimizations:

1. Warp-Specialized Pipelining: Splits threads inside an SM into producer warps (which fetch data from HBM using TMA) and consumer warps (which feed tensor cores), eliminating warp synchronization barriers entirely.
2. Ping-Pong GEMM Scheduling: Alternates matrix multiplications between different tensor core units to hide memory load latencies completely.
3. Asynchronous Softmax Overlap: Executes the exponential and reduction operations of attention Softmax concurrently with next-block tensor multiplications.

On an NVIDIA H100, FlashAttention-3 achieves up to 75% of theoretical peak tensor core utilization (over 700 TFLOPS), compared to just 35% utilization on naive attention implementations.

Mixture-of-Experts (MoE) Benchmarks: DeepSeek V3 & Mixtral 8x22B

Modern frontier architectures have shifted from dense models to Mixture-of-Experts (MoE). In an MoE model, only a fraction of total parameters (experts) are activated for each token. However, all expert weights must reside in VRAM.

We benchmarked serving Mixtral 8x22B (39B active parameters, 141B total parameters) across three GPU cluster configurations:

Cloud Rental Economics: The Total Cost of Ownership (TCO) Equation

Engineers often suffer from 'sticker shock' when viewing H100 hourly rates ($2.49 to $3.85/hr) compared to A100 rates ($1.64 to $2.20/hr). However, when evaluated on a Work-Completed-per-Dollar basis, the H100 and H200 are consistently the cheaper options for modern workloads.

The Mathematical Breakdown

Consider training a model that requires 100,000 steps on an 8-GPU cluster:

# 8x A100 Cluster:
Step Time: 1.48 seconds
Total Run Time: 100,000 * 1.48s = 148,000s = 41.1 hours
Hourly Cost: 8 * $1.89/hr = $15.12/hr
Total Cost on A100: 41.1 hrs * $15.12/hr = $621.43

# 8x H100 Cluster:
Step Time: 0.495 seconds
Total Run Time: 100,000 * 0.495s = 49,500s = 13.75 hours
Hourly Cost: 8 * $2.69/hr = $21.52/hr
Total Cost on H100: 13.75 hrs * $21.52/hr = $295.90

NET SAVINGS WITH H100: $325.53 (52.4% Cheaper!)
TIME SAVED: 27.35 Hours faster!

This mathematical reality holds true across almost all Transformer-based workloads due to the Hopper Transformer Engine. Renting an A100 is only cheaper when your job cannot utilize FP8 precision or when your workload is bounded by CPU data preprocessing rather than GPU compute.

Hardware Selection Guide: When to Rent Which Chip

Choose NVIDIA A100 (80GB) When:

• You are performing parameter-efficient fine-tuning (PEFT / LoRA / QLoRA) on smaller 7B–14B models where compute is not the bottleneck.
• Your legacy codebase relies on custom CUDA C++ kernels that have not been recompiled or validated for Hopper’s SM 90 architecture.
• You are doing computer vision (YOLOv10, ResNet), stable diffusion image rendering, or audio processing where FP8 is not supported.
• Check out our interactive GPU comparison table to find lowest A100 prices.

Choose NVIDIA H100 (80GB SXM5) When:

• You are training or full-parameter fine-tuning models larger than 30 billion parameters.
• You need high-throughput production LLM inference using vLLM or TensorRT-LLM with native FP8 quantization.
• You are deploying multi-node distributed clusters that require 3.2 Tbps InfiniBand interconnects.
• Explore availability at CoreWeave, Lambda, and Runpod.

Choose NVIDIA H200 (141GB) When:

• You want to serve large 70B models with long context windows (32K to 128K tokens) on fewer physical GPUs.
• You need to avoid multi-GPU tensor parallelism latency penalties on inference.
• Your workload is severely memory-bandwidth bound (e.g. mixture-of-experts models like Mixtral 8x22B or DeepSeek V3).

Choose Blackwell B200 (192GB) When:

• You are deploying frontier-class foundation models at hyperscale with high concurrent client request volumes.
• You can leverage native FP4 quantization for ultra-fast decode speeds.
• Your budget permits early access premium pricing on hyperscalers like Azure and OCI.

Form Factor Realities: SXM5 vs. PCIe vs. NVL

When selecting Hopper H100 instances in cloud provider catalogs, you will encounter two primary form factors: H100 SXM5 and H100 PCIe. Understanding the physical differences is vital, as they are not performance equivalents:

• H100 SXM5 (Board-Mounted Mezzanine): The true flagship. Operates at a full 700-watt TDP, enabling sustained GPU boost clocks up to 1,980 MHz. Features full 4th-Gen NVLink with 900 GB/s bidirectional bandwidth across all cards. Required for all multi-GPU distributed training and high-concurrency LLM serving. Available at CoreWeave, Lambda, and Runpod.
• H100 PCIe (Standard Server Add-in Card): Constrained by standard dual-slot server chassis cooling and power delivery, the PCIe variant is throttled to 350 watts TDP (half the power of SXM5). Its theoretical FP8 tensor performance drops from 1,979 TFLOPS down to 1,513 TFLOPS. Furthermore, NVLink bandwidth is restricted to dual-bridge 600 GB/s or eliminated entirely. Renting H100 PCIe for multi-node training will deliver 30% to 45% slower iterations than SXM5.
• H100 NVL (Dual-GPU Card for LLMs): Specifically designed for LLM inference on PCIe servers. Pairs two GH100 dies on a single dual-slot board with 94GB of HBM3 per die (188GB total) and NVLink interconnecting the two chips. Excellent for mid-sized inference deployments where SXM5 chassis are unavailable.

Multi-Instance GPU (MIG): Fractional Slicing for Cost Efficiency

Both the Ampere A100 and Hopper H100 support NVIDIA Multi-Instance GPU (MIG) technology. Unlike software-level time-slicing where multiple processes share the same CUDA cores and fight for memory bandwidth, MIG physically partitions the GPU at the hardware level into up to seven independent GPU instances.

Each MIG slice possesses its own dedicated high-bandwidth memory controllers, cache, and SM compute clusters. A fault or out-of-memory crash in one MIG slice has zero impact on neighboring slices running on the same physical chip.

CLI Diagnostics: Verifying Hopper Architecture in the Cloud

When provisioning an H100 or H200 node, verify your hardware capabilities with these low-level commands:

# 1. Query full GPU details and verify SXM5 vs PCIe form factor
nvidia-smi -q | grep -E "Product Name|Product Architecture|Compute Mode|GPU Part Number"

# 2. Check NVLink 4 link status and bandwidth per port
nvidia-smi nvlink -s
nvidia-smi nvlink -c

# 3. Inspect Memory Clock and PCIe link generation
nvidia-smi --query-gpu=gpu_name,driver_version,pcie.link.gen.current,pcie.link.width.current,memory.total,memory.free --format=csv

# 4. Run PyTorch to verify Transformer Engine FP8 availability
python3 -c "import torch; print('CUDA available:', torch.cuda.is_available()); print('Device:', torch.cuda.get_device_name(0)); print('Compute Capability:', torch.cuda.get_device_capability(0))"
# On Hopper, Compute Capability must report (9, 0)
# On Blackwell, Compute Capability will report (10, 0)

Mixture-of-Experts (MoE) Architecture: Why H200 and B200 Are Mandatory

The artificial intelligence industry has undergone an architectural migration from dense Transformer models (like Llama 2) to sparse Mixture-of-Experts (MoE) models (such as Mixtral 8x22B, DBRX, and DeepSeek-V3). MoE models decouple total parameter count from compute required per token: an MoE model might contain 140 billion parameters in memory, but only activate 39 billion parameters per forward pass.

This architectural shift fundamentally shifts the performance bottleneck from raw FLOPS to High-Bandwidth Memory (HBM) capacity and memory bandwidth. During the generation (decode) phase of an LLM, every active weight must be fetched from HBM into the compute registers for every single generated token. When serving an MoE model:

• On an A100 (80GB at 2.0 TB/s): The GPU memory bus is saturated immediately. Token generation slows down dramatically (often dropping to 12–18 tokens/second per user), while 70% of the CUDA cores sit idle waiting for memory reads.
• On an H100 (80GB at 3.35 TB/s): The increased bandwidth provides a 1.6x speedup in weight streaming, but fitting large MoE weights still mandates aggressive pipeline parallelism across multiple nodes, incurring networking overhead.
• On an H200 (141GB at 4.8 TB/s): A single node of 8x H200 provides 1,128 GB of combined HBM3e memory. This allows teams to fit massive 200B+ MoE checkpoints entirely within a single NVLink server chassis, achieving decode speeds exceeding 85 tokens/second per stream at fraction of the infrastructure cost.

Enterprise Cluster Sizing & TCO Decision Framework

Summary & Related Resources

Understanding GPU architecture ensures your team does not overpay for compute while sitting on underutilized memory bandwidth. For detailed provider reviews and live pricing matrices, check out:

• Top 10 GPU Cloud Providers Ranked
• Dedicated H100 GPU Cloud Server Guide
• Cheapest GPU Cloud Server: Cost Traps & Math
• RTX 4090 vs A100 & H100 Cloud Reality