See how AI is used and how it is running, without shipping prompts to a vendor.
Hosted observability tools for LLMs work by collecting traces, and LLM traces contain prompts, retrieved documents, and responses. Control collects usage, performance, and model health telemetry inside the runtime, so operations teams get the view they need and sensitive content stays where it is.
Telemetry is data too.
Debugging an AI application usually means looking at what was sent and returned. That is reasonable inside your boundary and a data-handling problem outside it: a trace exported to a third-party service is a copy of regulated content in a system your data map may not include.
Control separates the two. Metrics such as token counts, latency, error rates, and decision outcomes are content-free and can go to your existing observability stack. Content-bearing detail stays in the local audit log under the same access controls as the source data.
Four views of the same runtime.
| View | Signals | Who uses it |
|---|---|---|
| Usage | Requests and tokens by user, team, agent, application, and model; top datasets retrieved | Platform owners, finance, governance |
| Performance | Gateway latency, time to first token, generation throughput, retrieval latency, queue depth | Platform and SRE teams |
| Model health | Endpoint availability, error rates, GPU memory and utilization, model version in service | Infrastructure and ML teams |
| Governance | Allow, deny, and approval counts by policy; DLP denials by label and destination; pending approvals | Security and risk |
Operational signals for a private AI estate.
Per-identity usage
Attribute consumption to users, agents, and applications rather than one shared API key, for capacity planning and chargeback.
Capacity planning
Compare request volume with GPU utilization per model to see when to add hardware or move a workload to a smaller model.
Policy outcomes
A rise in denials after a policy change, or an agent suddenly hitting new datasets, is visible as it happens.
Version awareness
Every metric is tagged with the model version in service, so a regression after an upgrade is easy to isolate.
Monitoring that respects the boundary.
- Metrics are content-free by design
- Prompts and responses stay in the local audit log
- No telemetry is sent to app.data.co by default
- Works in air-gapped deployments
- Designed to feed the observability stack you already run
- Dashboards respect Control access roles
Signals worth an alert, and what they usually mean.
A starting set for a private AI deployment. Thresholds depend on your traffic and are best set after observing a normal baseline.
| Signal | Often indicates | First thing to check |
|---|---|---|
| Readiness check failing | A component is down or the runtime cannot enforce | Runtime status, license state, and policy set validity |
| License entering grace | Renewal has not been applied | Replace the license file and reload; enforcement continues during grace |
| Spike in denials for one agent | New untrusted content, a prompt change, or a scope gap | Decision log reasons for that identity |
| Denials across every request | An invalid policy set at startup, failing closed | Policy validation errors and test results |
| Growing pending approvals | Approver group not staffed or not notified | Queue age and approver coverage |
| Rising time to first token | Model saturated or GPU memory pressure | Queue depth and utilization for that endpoint |
| Blocked egress attempts | Something is trying to reach an unregistered destination | Recent egress attempts in runtime status |
What the runtime exposes about itself.
The runtime provides liveness and readiness endpoints that contain no secrets and no content, suitable for a load balancer or orchestrator. An administrative status view adds the license state, the active policy set version, the latest audit log position, and the egress guard's allowed destinations along with recent attempts to reach anything else. The license moves through valid, expiring, grace, and expired; during grace the runtime keeps enforcing and reports itself degraded, which gives operations a clear signal well before anything stops.
For connected deployments, an optional heartbeat can report runtime version, license status, health, component states, and policy set version to app.data.co. It uses a strict schema that rejects unknown fields, so nothing else can be attached to it, and it is refused entirely in air-gapped mode. Whether you enable it or not, the operational view you rely on is the one inside your environment.
Our team can watch the runtime around the clock.
Managed monitoring covers alerting and response for the Control runtime and its hardware, operated under your access rules and inside your environment.
Questions buyers ask
Does any telemetry go to Data.co?
Not by default. If you opt into fleet administration on app.data.co, only deployment-level operational status such as version and health is shared. Prompts, responses, and per-request records are never sent.
Can we use our existing monitoring tools?
That is the intent. Content-free metrics are designed to be consumed by the observability stack you already operate, so AI dashboards sit next to the rest of your infrastructure.
Does it monitor answer quality?
Control focuses on usage, performance, health, and governance outcomes. Quality evaluation depends on your use case and data; evaluation runs can be executed inside your environment against the same private models.
How is this different from the audit log?
The audit log is a per-request evidence record of identities, decisions, and policies. Monitoring is aggregated, content-free telemetry for operating the system. They share identifiers so you can move from a metric spike to the underlying records when you have access.
How do we know governance is actually running, not just the models?
Readiness reflects the enforcement path, not only the model servers. A runtime without a valid license or policy set fails closed and reports it, and the active policy set version is visible in status so you can confirm the expected rules are live.
Can monitoring show cost or capacity per team?
Usage is attributed to identities, so requests and tokens can be grouped by user, team, agent, or application. Converting that into cost depends on your hardware and internal chargeback model.
See it against your own use case.
A 30-minute technical walkthrough, policy simulation on your scenarios, and a deployment recommendation for on-prem, private cloud, or hybrid.