Data stays local. Compute can burst.
A hybrid deployment keeps source data, indexes, and the policy engine inside your facility, and adds GPU capacity elsewhere for workloads policy allows to move. The boundary is a set of explicit rules, not a network diagram nobody revisits.
Most workloads are not equally sensitive.
A typical organization has some workloads that must stay on-prem, such as retrieval over patient records or privileged matter files, and others that are less sensitive but compute-heavy, such as summarizing public filings, generating test data, or nightly batch classification. Sizing on-prem hardware for the peak of both is expensive.
Hybrid lets the gateway route by classification and purpose. Requests involving restricted data run on local GPUs. Requests that policy permits can run on additional capacity in a single-tenant cloud or on dedicated servers. The data sources and indexes do not move.
Where a request runs is a rule you can read.
alias: general-chat
pools:
- name: onprem-gpu
location: datacenter-east
- name: cloud-burst
location: private-cloud-us-central
rules:
- when: { classification: [PHI, privileged, restricted] }
route: onprem-gpu # never leaves the facility
- when: { purpose: batch, classification: [public, internal] }
route: cloud-burst
- default: onprem-gpu
fallback_across_boundary: falseIllustrative example · final API surface documented in /docs
A typical split.
Your split will reflect your classifications and contracts; this shows the common starting point.
| Component | On-premises | Additional compute |
|---|---|---|
| Source data and connectors | Yes | No |
| Vector indexes and documents | Yes | No |
| Policy engine and audit log | Yes | Enforcement point only |
| Inference on restricted data | Yes | No |
| Inference on permitted data | Optional | Yes |
| Batch jobs on public or internal data | Optional | Yes |
What makes hybrid work in practice.
- Private, encrypted link between sites
- Latency budget for interactive requests
- No automatic failover across the boundary
- Classification applied before routing
- One policy set, one audit log
- Capacity monitoring on both sides
Design for the day the link goes down.
The link between sites will eventually fail or degrade. Because the policy engine, indexes, and audit log stay on-prem, an outage does not stop governance or retrieval; it removes the additional compute. Requests that policy allows to run elsewhere then either queue, run on local GPUs if capacity exists, or return a clear error. Restricted requests are unaffected because they never depended on the link.
Latency is the quieter problem. An interactive request that retrieves locally, then sends prompt and passages across a wide-area link for generation, adds round-trip time and exposes it to congestion. Streaming responses help, but batch work is usually the better fit for remote capacity. Measure the link under realistic load before routing interactive traffic across it.
Deciding where a workload should run.
| Question | If yes | If no |
|---|---|---|
| Does the prompt include restricted classifications? | Local pool only | Eligible for either pool |
| Is it interactive and latency-sensitive? | Prefer local | Remote capacity is a good fit |
| Is the volume large and deadline-driven? | Candidate for remote batch pool | Local capacity may suffice |
| Does a contract restrict processing location? | Encode it in routing rules | Route by classification alone |
Adding a second pool to an existing deployment.
- 1
Classify first
Make sure datasets and connectors carry classifications, since routing decisions depend on them.
- 2
Stand up the pool
Deploy model servers in the private cloud or on dedicated servers, registered as endpoints with a location attribute.
- 3
Write routing and policy
Express which classifications and purposes may use the remote pool, and test the rules before activation.
- 4
Shift traffic gradually
Start with one batch workload, review audit records and latency, then widen.
Questions buyers ask
What happens if on-prem GPUs are saturated?
For restricted data, requests queue or are rejected per your settings; they do not fail over to the other side. For permitted data, routing rules can send overflow to additional capacity.
Do documents ever leave the facility?
Not by default. Retrieval runs on-prem. If a permitted request is routed elsewhere, only the prompt and retrieved passages for that request are sent, and only when policy allows that classification to cross.
Is hybrid harder to operate?
It adds a second location to monitor and a link to maintain. Policies, models, and audit stay in one control plane, which keeps the extra work mostly in networking and capacity planning.
Can we start on-prem and add hybrid later?
Yes. Adding a second pool and routing rules does not require changing applications, which call the same aliases through the gateway.
Is traffic between sites encrypted?
It should be, at two layers: a private, encrypted link between sites, and TLS from the gateway to the remote model endpoints. Remote pools should accept traffic only from your gateways.
Where is the audit record for a request that ran remotely?
In the same on-prem audit log as every other decision. The policy decision happens before routing, so the record exists regardless of where the model ran.
Size it before you buy it.
An infrastructure assessment maps your workloads, data constraints, and existing hardware to a deployment model and a bill of materials you can review.