Open-weight language models, served on your hardware.
Data.co Control deploys and operates model serving inside your environment: vLLM on GPU servers for production concurrency, Ollama for workstations and small teams. You choose the models; prompts and completions never leave the boundary.
Choose the model for the task, not for the vendor.
Open-weight model families such as Llama, Mistral, Qwen, and Gemma are published under licenses that allow self-hosting, each with its own terms. They range from small models that run on a single workstation GPU to large models that need several data-center GPUs. Most enterprises end up with a few: a capable general model for chat and drafting, a smaller fast model for classification and extraction, and an embedding model for retrieval.
Data.co does not lock you into a model. Each model is a registry entry with a version, serving configuration, and approval status. Applications call a stable alias through the gateway, so you can test a newer model, compare outputs, and promote it without changing application code.
What determines the hardware a model needs.
These are general engineering considerations, not guarantees. Actual requirements depend on the model architecture, serving configuration, and traffic. An assessment measures them for your workloads.
| Factor | Why it matters | Rule of thumb |
|---|---|---|
| Parameter count | Weights must fit in GPU memory (VRAM) to serve efficiently | Roughly 2 bytes per parameter at 16-bit precision, before overhead |
| Quantization | 8-bit and 4-bit formats shrink weights so larger models fit | Smaller footprint; validate output quality on your own tasks |
| Context length | Long prompts (RAG, documents) grow the KV cache per request | Budget VRAM for cache, not only for weights |
| Concurrency | Simultaneous users share memory and compute | vLLM batches requests; more users means more cache headroom |
| Latency target | Interactive chat and batch jobs have different needs | Separate interactive and batch traffic where possible |
Two runtimes, one control plane.
vLLM for production
Continuous batching and paged attention for multi-user throughput on GPU servers, with tensor parallelism across GPUs.
Ollama for smaller footprints
Simple model serving for workstations, labs, and pilots where a single GPU or Apple silicon is enough.
Same gateway either way
Both runtimes sit behind the AI gateway, so identity, quotas, policy, and audit behave identically.
Health and capacity
Per-model latency, queue depth, and GPU memory are reported to monitoring, so you can see when to add capacity.
A model is a reviewed configuration, not a container someone started.
Serving parameters live in the registry alongside the model version, so the configuration that was approved is the configuration that runs.
alias: general-chat
model: llama-3.3-70b-instruct
weights:
source: registry://mirror/llama-3.3-70b-instruct
sha256: "<pinned digest>"
runtime: vllm
serving:
quantization: fp8
tensor_parallel: 2
max_model_len: 32768
gpu_memory_utilization: 0.90
status: approved
approved_by: ml-platform-reviewIllustrative example · final API surface documented in /docs
Nothing in the inference path phones home.
- Model weights mirrored and pinned by digest
- Prompts and completions processed locally
- No outbound calls from serving nodes required
- Usage metrics stay in your monitoring stack
- Model licenses recorded in the registry
- Works on air-gapped networks with offline media
A back-of-envelope estimate before any benchmark.
Weights are the floor. Multiply the parameter count by bytes per parameter: about 2 bytes at 16-bit precision, about 1 byte at 8-bit, and roughly half a byte at 4-bit, plus some overhead for quantization scales. By that arithmetic, a 70-billion-parameter model needs on the order of 140 GB for weights at 16-bit, around 70 GB at 8-bit, and somewhere near 35 to 40 GB at 4-bit. An 8-billion-parameter model is roughly a tenth of each figure.
The KV cache comes next. For every token in flight, the model stores keys and values for each layer, so cache memory grows with context length multiplied by the number of concurrent sequences. A deployment serving long RAG prompts to many users at once can need as much memory for cache as for weights. Architectures differ, and features such as grouped-query attention reduce cache size, which is why these estimates are a starting point for measurement, not a substitute for it.
A runbook for serving nodes.
Model serving is ordinary infrastructure with a few unusual failure modes. These are the routines that keep it predictable.
- 1
Pin the stack
Record GPU driver, CUDA or equivalent runtime, container image, and serving runtime versions together. Upgrade them as a set, never one at a time on a live node.
- 2
Watch the right signals
Track time to first token, generation throughput, queue depth, KV cache utilization, GPU memory, temperature, and error rates per alias, not only host CPU and disk.
- 3
Drain before maintenance
Remove a node from its pool, let in-flight requests finish, then patch or reboot. Large models take time to load, so plan warm-up before returning the node.
- 4
Canary model changes
Serve a new model version or serving configuration on a non-production alias first, compare outputs on your evaluation set, then promote through the registry.
- 5
Keep weights recoverable
Weights live in your artifact storage, verified by digest. Back up that store and the registry metadata so a failed node can be rebuilt without a fresh download.
Not every task needs the largest model you can host.
| Task | Typical model choice | Consideration |
|---|---|---|
| Classification, routing, extraction | Small instruction-tuned model | Low latency and cost per request; easy to run many in parallel |
| Drafting and summarization | Mid-sized general model | Quality gains flatten for many internal documents; test before upsizing |
| Grounded Q&A over documents | Mid-sized model plus strong retrieval | Retrieval quality often matters more than model size |
| Multi-step reasoning and code | Largest model the hardware allows | Most sensitive to quantization and context limits |
| Embeddings for RAG | Dedicated embedding model | Changing it later means re-indexing every collection |
Questions buyers ask
Which open-weight models can we run?
Models supported by vLLM or Ollama can generally be registered and served, which covers the widely used open-weight families. We help you shortlist candidates, check their license terms, and evaluate them on your own examples before production.
Does quantization hurt quality?
It can, and the effect varies by model and task. 8-bit formats are often close to full precision; 4-bit formats save more memory with a larger risk of degradation on demanding tasks. We recommend evaluating quantized and full-precision variants on your real prompts before choosing.
Can we fine-tune models?
Fine-tuned weights, including adapters, can be registered and served like any other model version, subject to the same approval workflow. Fine-tuning itself needs more GPU memory than inference, so it is sized separately in an assessment.
How do we update to a newer model?
Register the new version, test it against your evaluation set, and promote it to the alias your applications call. Rollback is a re-promotion of the previous version.
Can we run models on hardware we already own?
Often, yes. An assessment checks your existing GPUs, memory, drivers, and networking against the models and concurrency you need, and tells you what, if anything, to add.
Should interactive and batch traffic share the same GPUs?
It works at small scale, but a large batch job can fill the request queue and slow interactive users. Separate pools, or quotas at the gateway that cap batch throughput, keep chat responsive while still using spare capacity.
Can we serve a long context window for every request?
You can configure a long maximum context, but each long request reserves more KV cache, reducing how many requests fit at once. Many teams set a moderate default and route long-document work to a separate alias sized for it.
See it against your own use case.
A 30-minute technical walkthrough, policy simulation on your scenarios, and a deployment recommendation for on-prem, private cloud, or hybrid.