Restores you have tested, not backups you hope work.
We back up the parts of a private AI environment that are hard to rebuild: policies, audit logs, model registry, vector indexes, and configuration. Restores are tested on a schedule, and recovery objectives are set per contract.
Not everything needs the same treatment.
Some components can be regenerated; others cannot. The backup plan reflects the difference.
| Component | Rebuildable? | Approach |
|---|---|---|
| Policies and configuration | Partly, if in Git | Versioned and backed up |
| Audit logs | No | Backed up with retention you set; integrity preserved |
| Model registry metadata | No | Backed up with configuration |
| Model weights | Yes, from your mirror | Mirror protected; digests recorded |
| Vector indexes | Yes, by re-indexing, but slowly | Backed up to shorten recovery |
| PostgreSQL data | No | Point-in-time capable backups |
RPO and RTO are contract terms, not marketing numbers.
How much data you can afford to lose (recovery point objective) and how long you can be without the service (recovery time objective) depend on how central AI is to your operations and what you are prepared to spend on redundancy. We set them with you per contract, then design backup frequency, storage location, and standby capacity to meet them.
Re-indexing large document collections can take a long time, which is why vector indexes are backed up even though they can technically be rebuilt. GPU capacity for recovery is planned explicitly, since replacement hardware may not be quickly available.
A restore test, start to finish.
- 1
Plan
Agree which components are restored, to where, and what success looks like.
- 2
Restore
Recover into an isolated environment from the latest backups.
- 3
Verify
Check policies, audit log integrity, registry state, and retrieval results against the source.
- 4
Report
Record time taken, issues found, and fixes, compared with the contracted objectives.
Backing up evidence without breaking the chain.
The Control runtime writes its audit log as a hash chain: each record includes the hash of the one before it, so altering, deleting, or reordering any record is detectable. Backups have to preserve that property. Rotated segments are immutable once closed and are copied as they appear; the active file and the approvals store are snapshotted on the agreed schedule.
Verification starts from the first segment, so segments are archived as a complete set rather than pruned piecemeal. We also record the latest sequence number and hash somewhere separate from the log, such as your ticketing system or write-once storage, so a wholesale rewrite of the log would be detectable too. Restore tests run the runtime's audit verification against the restored copy and compare its head with that separate record.
What recovery looks like for common failures.
| Failure | Recovery path | Checked before returning to service |
|---|---|---|
| Failed disk on a database host | Replace and restore from point-in-time backup | Row counts, registry state, recent policy decisions |
| Corrupted vector index | Restore the index backup, then incremental re-sync | Retrieval evaluation set and permission filters |
| Bad policy activation | Previous policy set retained by the runtime; roll back the bundle | Policy tests and a sample of recent decisions |
| Lost GPU node | Shift aliases to remaining capacity or standby hardware | Serving health and latency per alias |
| Loss of the primary site | Standby environment, if contracted | Full restore test checklist |
How backups are handled.
- Backups stay in locations you approve
- Encrypted with keys you control
- Separate credentials from production
- Immutable or write-once storage where available
- Restore tests on a contracted schedule
- Air-gapped backups stay inside the isolated network
Questions buyers ask
What RPO and RTO do you offer?
They are set per contract based on your requirements and the redundancy you choose to fund. We design and test against the objectives you agree to.
Do backups leave our environment?
Only to locations you approve, such as a second site or your own storage. In air-gapped deployments, backups stay within the isolated network.
How often are restores tested?
On a schedule defined in the contract, with results reported each time. Tests restore into an isolated environment so production is not affected.
Can audit logs be tampered with during restore?
Audit logs are append-only in production and their integrity is checked during restore tests, so gaps or alterations would surface in the verification step.
Does this cover disaster recovery for the whole site?
It can, with a standby environment at a second site. Scope, including whether standby GPU capacity is kept warm, is defined per contract.
Can Data.co read our backups?
Backups are encrypted with keys you control and stored where you approve. Our engineers operate the backup jobs through the access you grant; reading backup contents is not part of routine work.
See it against your own use case.
A 30-minute technical walkthrough, policy simulation on your scenarios, and a deployment recommendation for on-prem, private cloud, or hybrid.