A practical framework for choosing GPUs for internal AI infrastructure — from A100 and H100 to H200 and B200 — based on real workload, memory bandwidth, and company size, not just VRAM.
Many companies want to bring AI infrastructure in-house.
The reason is simple: private data, lower long-term dependency on external APIs, more control, and the ability to deploy internal AI assistants, RAG systems, computer vision models, fine-tuning pipelines, and LLM inference servers.
But choosing the right GPU is not simple.
A startup should not buy GPUs the same way a big company does. A research team should not buy GPUs the same way a production AI platform team does.
And most importantly: for AI workloads, GPU memory size is not enough. You need to understand Tensor Cores, memory bandwidth, interconnect, software support, cooling, and the real workload.
TL;DR — key takeaways
- VRAM is a floor, not a ranking. Enough memory to hold the model is a hard requirement; past that, memory bandwidth usually decides how fast you actually go. - Match the GPU to the workload, not the benchmark. Inference, fine-tuning and full training have different bottlenecks and different right answers. - A100 is still a strong budget enterprise option. H100 is the modern default for training and heavy inference. H200 wins when memory capacity and bandwidth are the constraint. B200/GB200 is for organisations building AI factories. - L40S and RTX 6000 Ada are the practical picks for small teams running internal RAG, computer vision or moderate inference on-premise. - Interconnect matters the moment you need more than one GPU. NVLink versus PCIe changes what multi-GPU training is possible at all. - The right question is cost-to-run-your-workload-reliably, not raw peak performance.
First: do not choose a GPU only by VRAM
The first mistake companies make is asking:
"How much GPU memory do we need?"
That is important, but incomplete. For AI, you should also ask:
- What type of models will we run? - Will we train, fine-tune, or only do inference? - How many users will use the system at the same time? - How long is the context window? - Do we need multi-GPU training? - Do we need low latency or high throughput? - Do we need enterprise reliability and support?
A GPU with large memory but weak bandwidth can bottleneck. A GPU with strong compute but not enough memory can fail. A GPU with good raw specs but weak software support can slow down the engineering team.
So the right GPU is not always the biggest GPU. The right GPU is the one that matches the workload.
The core types companies should understand
When people compare GPUs, they often talk about "cores." But not all cores are equally important for AI.
CUDA cores
CUDA cores are general-purpose parallel compute units. They are useful for many GPU operations, including preprocessing, custom kernels, simulation, image operations, and non-tensor workloads.
But modern deep learning is not accelerated mainly by CUDA cores.
Tensor Cores
Tensor Cores are specialized units designed to accelerate matrix operations. This matters because deep learning is mostly matrix multiplication.
Training transformers, running LLM inference, using FP16, BF16, TF32, FP8, INT8, or INT4 — all of this depends heavily on Tensor Core performance.
For AI companies, Tensor Cores are usually more important than CUDA cores.
But even that is not enough. You should not simply buy the GPU with the highest Tensor Core number. You also need enough HBM memory, enough memory bandwidth, and good software support.
RT Cores
RT Cores are mainly for ray tracing and rendering. They matter for graphics, simulation, 3D, digital twins, and rendering workloads.
For pure LLMs, RAG, NLP, and most deep learning workloads, RT Cores are not the main factor.
The hidden bottleneck: memory bandwidth
For LLM inference, memory bandwidth is often as important as raw compute.
When a model generates tokens, the GPU constantly moves data between memory and compute units. If memory cannot feed the compute units fast enough, the Tensor Cores wait. That means the GPU may look powerful on paper but still underperform in real inference.
This is why data center GPUs use HBM memory instead of normal GDDR memory. HBM provides much higher bandwidth and is designed for heavy AI and HPC workloads.
For example, NVIDIA H200 offers 141 GB of HBM3e memory at 4.8 TB/s, which is nearly double the H100 memory capacity and 1.4× more memory bandwidth according to NVIDIA.
A100 vs H100 vs H200: what is the difference?
| GPU | Memory highlight | Best for | |---|---|---| | A100 | Large HBM2e, over 2 TB/s bandwidth | Budget-friendly enterprise AI, MIG-shared workloads | | H100 | 4th-gen Tensor Cores + FP8 Transformer Engine | Serious LLM training and inference at scale | | H200 | 141 GB HBM3e @ 4.8 TB/s | Memory-bound / long-context LLM serving | | DGX B200 (8-GPU) | 1,440 GB total / 64 TB/s total HBM3e | Frontier-scale training and serving | | L40S | 48 GB GDDR6 ECC @ 864 GB/s | Workstation-class inference, CV, prototyping | | RTX 6000 Ada | 48 GB, 4th-gen Tensor Cores | AI workstations, developer machines | | AMD MI300X | 192 GB HBM3, 5.3 TB/s peak per GPU | Memory-heavy inference, CUDA-alternative stacks |
NVIDIA A100
The A100 is older but still very useful. It is based on the Ampere architecture and was built for AI, data analytics, and HPC. NVIDIA describes it as a Tensor Core GPU with large memory, over 2 TB/s of memory bandwidth, NVLink/NVSwitch scalability, and support for Multi-Instance GPU partitioning.
A100 is good for:
- Startups with limited budget - Internal RAG systems - Fine-tuning smaller models - Computer vision training - Batch inference - Shared GPU environments using MIG
The main advantage of A100 today is cost efficiency, especially if bought as used or refurbished enterprise hardware. The main limitation is that it is not as strong as H100/H200 for modern LLM workloads, especially FP8 transformer workloads.
NVIDIA H100
The H100 is a much stronger AI GPU. It is based on the Hopper architecture and includes fourth-generation Tensor Cores and a Transformer Engine with FP8 precision. NVIDIA says H100 can provide major speedups for large transformer training compared with the previous generation.
H100 is good for:
- Serious LLM inference - Large-scale fine-tuning - Multi-GPU training - High-throughput production AI - Companies building internal AI platforms
Compared with A100, H100 is much better for modern LLM workloads because it supports FP8 acceleration and has stronger Tensor Core performance. But it is expensive and requires proper server infrastructure.
Buying H100 is not only buying a GPU. It usually means buying the right server, cooling, power, networking, storage, and support.
NVIDIA H200
The H200 is not just "a faster H100." The most important difference is memory.
H200 is based on Hopper like H100, but it brings much larger and faster HBM3e memory: 141 GB at 4.8 TB/s.
H200 is good for:
- Large LLM inference - Long-context inference - RAG systems with large context windows - Bigger batch sizes - Memory-heavy workloads - Serving larger models with fewer GPUs
If your bottleneck is memory capacity or memory bandwidth, H200 is more attractive than H100. For many LLM serving workloads, this matters a lot because inference is often memory-bound.
What about B200 and Blackwell?
If a large company is buying new infrastructure in 2026 and has a serious AI budget, it should also consider Blackwell GPUs such as B200 or GB200 systems.
NVIDIA's DGX B200 system uses eight Blackwell GPUs with 1,440 GB total GPU memory and 64 TB/s total HBM3e bandwidth, while NVIDIA's Blackwell architecture adds newer Tensor Core capabilities including FP4 support for AI workloads.
B200/GB200 systems are for companies building an AI factory, not just running a simple chatbot. They are more relevant for:
- Large-scale model training - Frontier model inference - High-volume AI serving - Multi-team enterprise AI platforms - Large MoE workloads
For a startup, this is usually too expensive. For a big company, it may make sense if AI is core to the business.
Where L40S and RTX 6000 Ada fit
Not every company needs A100, H100, or H200. For smaller internal AI workloads, NVIDIA L40S or RTX 6000 Ada can be more realistic.
The L40S has 48 GB GDDR6 ECC memory, 864 GB/s memory bandwidth, 18,176 CUDA cores, and 568 fourth-generation Tensor Cores. RTX 6000 Ada also provides 48 GB of graphics memory with fourth-generation Tensor Cores and CUDA cores for AI, graphics, and compute workloads.
These GPUs are good for:
- AI workstations - Computer vision - Medium-size model inference - Small fine-tuning jobs - Prototype RAG systems - Internal demos - Developer machines
They are not the same class as H100/H200 for large LLM training, but they can be very useful for startups that need practical internal AI without building a full data center.
NVIDIA is not the only option
NVIDIA is dominant because of CUDA, cuDNN, TensorRT, vLLM support, PyTorch maturity, and the ecosystem around AI deployment.
But AMD is becoming more relevant for enterprise AI. For example, AMD MI300X offers 192 GB HBM3 memory and high memory bandwidth, making it attractive for memory-heavy AI inference workloads. AMD's data sheet lists 1.5 TB HBM3 memory across an 8-GPU MI300X platform and 5.3 TB/s max peak theoretical memory bandwidth per GPU.
The tradeoff is software maturity. If your team is already optimized around CUDA and NVIDIA tools, switching to AMD may increase engineering complexity. If your company has a strong infrastructure team and wants more vendor flexibility, AMD can be worth evaluating.
GPU choice by company size
Small startup
A small startup should usually not start with H100 or H200 unless AI infrastructure is the product itself.
Better choices:
- RTX 4090 / RTX 5090 class GPUs for experiments - RTX 6000 Ada for professional workstation use - L40S for internal inference and computer vision - Used A100 40GB/80GB if enterprise reliability matters
Best strategy: start small, measure real usage, then upgrade. A startup should focus on experiments, prototypes, RAG systems, LoRA fine-tuning, and small production inference. Do not buy a cluster before proving the workload.
Growing startup or mid-size company
A growing AI company may need stronger internal infrastructure.
Better choices:
- A100 80GB for cost-efficient training and inference - L40S for mixed AI and visual workloads - H100 if training and high-throughput inference are important - H200 if LLM serving and long context are the bottleneck
This type of company should think seriously about GPU scheduling, Docker/Kubernetes, model serving, monitoring, user concurrency, security, and data governance. At this stage, the GPU is not the full system — the platform matters.
Big company or enterprise
A big company should not only buy GPUs. It should design an internal AI platform.
Better choices:
- H100/H200 HGX servers - DGX H100/H200 systems - B200/GB200 systems for very large AI workloads - AMD MI300X if the company can support ROCm
Enterprise workloads usually need multi-GPU servers, NVLink/NVSwitch, InfiniBand or high-speed Ethernet, shared storage, GPU monitoring, access control, job scheduling, MLOps, security, and support contracts.
For big companies, the real question is not:
"Which GPU is fastest?"
The real question is:
"Which infrastructure gives us reliable AI capacity for many teams?"
The most practical decision rule
Here is the simplest way to think about it, as a rough heuristic:
In words:
- For experimentation: use workstation GPUs. - For small production inference: use L40S, RTX 6000 Ada, or A100. - For serious LLM inference: use H100 or H200. - For memory-heavy LLM serving: prefer H200 over H100. - For frontier-scale AI: consider B200/GB200 or equivalent rack-scale systems. - For budget-conscious enterprise AI: consider used A100 or AMD MI300X, depending on your software stack.
Do not forget the hidden costs
Internal AI infrastructure is not only GPU cost. You also need:
- Server chassis - CPU and RAM - NVMe storage - Networking - Cooling - Electricity - Rack space - Monitoring - Security - Backup - Maintenance - Driver management - MLOps tools - Engineering time
High-end GPUs also consume serious power. NVIDIA's Blackwell comparison shows H100/H200-class chips can reach high TGP levels, while B200-class GPUs go even higher, which makes power and cooling part of the buying decision.
A cheap GPU can become expensive if it creates downtime. An expensive GPU can be wasteful if your workload does not use it.
Final recommendation
For AI, companies should care about Tensor Cores. But they should not care only about Tensor Cores.
The best GPU decision depends on six things:
1. Model size 2. Memory capacity 3. Memory bandwidth 4. Tensor Core generation 5. Multi-GPU interconnect 6. Software ecosystem
A100 is still a strong budget enterprise option. H100 is a powerful modern training and inference GPU. H200 is better when memory capacity and bandwidth are the bottleneck. B200/GB200 is for companies building serious AI factories. L40S and RTX 6000 Ada are practical options for smaller teams and workstations.
The right question is not:
"What is the most powerful GPU?"
The right question is:
"What is the cheapest GPU infrastructure that can run our real AI workload reliably?"
That is how startups and big companies should choose internal AI hardware.
Frequently asked questions
How much GPU memory do I need to run an LLM?
As a rough floor, a model in 16-bit precision needs about two gigabytes of VRAM per billion parameters, plus headroom for the KV cache, which grows with batch size and context length. Quantising to 8-bit roughly halves that, and 4-bit halves it again at some quality cost. Size for your longest realistic context and largest concurrent batch, not for the model weights alone — running out of memory mid-request is a much worse failure than being slightly slower.
Is the H100 worth it over the A100 for a small company?
Often not. The A100 remains capable and is considerably cheaper on the second-hand and cloud markets. The H100 earns its price when you are training or fine-tuning frequently, serving high-concurrency inference, or benefiting from FP8 support. If your workload is internal RAG over company documents with modest traffic, an A100 — or an L40S — will usually do the job for far less money.
What is the difference between the H100 and the H200?
They share the same compute architecture. The H200 carries substantially more memory and considerably higher memory bandwidth. That matters for workloads that are memory-bound rather than compute-bound: serving very large models, long-context inference, and large-batch decoding. If you are not hitting a memory wall on an H100, the H200 will not feel dramatically faster.
Should a startup buy GPUs or rent from the cloud?
Rent until your utilisation is consistently high. Owned hardware only wins when the GPUs are busy most of the time; a bought GPU sitting idle is pure loss, while a rented one simply stops billing. The usual pattern is to prototype and burst in the cloud, then buy once you have a steady, predictable workload and a data-residency or cost reason to bring it in-house.
Do I need NVLink for multi-GPU AI training?
For inference across two GPUs, usually not — PCIe is generally adequate. For training or fine-tuning large models across many GPUs, interconnect becomes the bottleneck fast, because gradients have to be synchronised every step. If multi-GPU training is your main workload, treat interconnect as a first-class selection criterion rather than a specification you check afterwards.
What GPU is best for computer vision and medical imaging models?
Vision models are typically far smaller than LLMs, so memory capacity is rarely the constraint — throughput and data-loading are. An L40S or RTX 6000 Ada handles most CNN and detector training comfortably. The chest X-ray triage pipeline I built uses a ResNet50 classifier and a YOLOv8 detector, neither of which requires datacentre-class memory; the practical bottleneck there was image preprocessing throughput, not the GPU.
Related reading
- AI Agents from Prototype to Production: What Actually Breaks — what runs on the hardware, and what breaks when it does. - Two-Stage Chest X-Ray Triage: ResNet50 + YOLOv8 at 96.9% Accuracy — a real computer-vision workload and its actual resource profile. - RAG vs Fine-Tuning vs Prompt Engineering: How to Actually Choose — the decision that determines whether you need training hardware at all.