Inference nodes sized for the models you will actually run.
GPU servers carry production model serving: vLLM inference for many concurrent users, embedding generation for RAG, and fine-tuning when you need it. We specify them from your workloads, not from a catalog.
What drives the GPU configuration.
General considerations, not guarantees. Real requirements depend on the specific model, runtime version, and traffic pattern, which is why we measure during an assessment.
| Consideration | Effect on hardware | Trade-off |
|---|---|---|
| Model size | Weights must fit in combined GPU memory; about 2 bytes per parameter at 16-bit | Larger models need more GPUs or quantization |
| Quantization (8-bit, 4-bit) | Reduces weight memory substantially | Possible quality loss; evaluate on your tasks |
| Context length | KV cache grows with tokens per request | Long-context RAG needs VRAM headroom beyond weights |
| Concurrent users | More simultaneous sequences need more cache | More GPUs, or queueing at peak |
| Multi-GPU serving | Tensor parallelism splits a model across GPUs | Interconnect bandwidth affects performance |
| Fine-tuning | Training needs gradients and optimizer state | Usually more memory than inference for the same model |
Memory first, then everything else.
For LLM inference, GPU memory is usually the first constraint. A model that does not fit must be quantized, split across more GPUs, or replaced with a smaller one. After the weights fit, the remaining memory determines how many requests and how much context can be served at once.
We generally recommend separating interactive and batch serving where budgets allow, keeping headroom for model upgrades over the hardware's life, and validating a quantized model's output quality before sizing the fleet around it.
What comes configured.
Driver and runtime stack
GPU drivers, container runtime, and serving software validated together.
Burn-in
Hardware stress-tested before shipment to surface early failures.
Control integration
Nodes join your Control deployment and report health to monitoring.
Documentation
As-built configuration, firmware versions, and support contacts.
Questions the assessment answers.
- Which models, at which precision
- Peak concurrent users and context lengths
- Interactive versus batch split
- Whether fine-tuning is in scope
- Rack power and cooling available
- Redundancy needed for availability
The rest of the node still matters.
A server with capable GPUs can still underperform if the supporting components are an afterthought.
| Component | Why it matters for AI serving | What to consider |
|---|---|---|
| GPU interconnect | Tensor parallelism exchanges data between GPUs on every token | Direct high-bandwidth links between GPUs help multi-GPU models; plain PCIe can bottleneck |
| System memory | Model loading, CPU-side batching, and embedding pipelines use host RAM | Comfortably more than the model's weight size eases loading and swapping |
| Local NVMe | Large weight files load from disk at startup and failover | Fast local storage shortens restarts and model swaps |
| CPU cores | Tokenization, request handling, and document parsing run on the CPU | Ingest-heavy RAG nodes need more cores than pure inference nodes |
| Network | Multi-node serving and index builds move a lot of data | Separate data and management networks; plan for out-of-band access |
| Power and cooling | Dense GPU nodes draw far more than typical servers | Confirm per-rack power and cooling before choosing a chassis |
Keeping GPU nodes reliable over their life.
- 1
Monitor hardware health
Watch GPU memory error counters, temperature, power draw, and clock throttling. Rising correctable errors or thermal throttling are early signs worth acting on.
- 2
Pin driver versions
Test driver and runtime upgrades on one node, validate model output and throughput, then roll out. Keep the previous versions available for rollback.
- 3
Drain, then service
Take a node out of its serving pool before firmware updates or part replacement so in-flight requests complete.
- 4
Plan the next model
Newer models tend to be larger or use longer contexts. Leaving memory headroom at purchase extends the useful life of the hardware.
Questions buyers ask
How many GPUs do we need?
It depends on model size, precision, concurrency, and context length. A small pilot may need one GPU; production RAG for a large department may need several across nodes. We size from measured workloads rather than estimates.
Which GPU vendors do you support?
We specify hardware supported by the serving runtimes, which today primarily means NVIDIA data-center GPUs, with other accelerators considered where vLLM support is mature for your models.
Can one server run several models?
Yes, if memory allows. Smaller models can share a GPU, and larger models can be assigned dedicated GPUs. The serving configuration is recorded in the model registry.
Do you quote prices on this page?
No. GPU pricing and availability change frequently. You receive a current quote after an assessment defines the configuration.
Can we fine-tune on the same servers we use for inference?
Possibly, if scheduled outside peak inference hours and memory allows. Many teams keep separate capacity so training jobs do not affect interactive users.
Should we buy one large node or several smaller ones?
Several nodes give you maintenance headroom and failure tolerance; one large node can be simpler and suits models that need many GPUs with fast interconnect. The right answer depends on the largest model you must serve and how much downtime you can accept.
Do GPU servers also run the embedding model for RAG?
Usually. Embedding models are small relative to chat models and can share a GPU, but large initial ingests or re-indexing runs are heavy, so schedule them away from peak interactive hours or give them dedicated capacity.
Size it before you buy it.
An infrastructure assessment maps your workloads, data constraints, and existing hardware to a deployment model and a bill of materials you can review.