Answers grounded in your documents, limited to what each user may see.
Private RAG indexes your documents and records inside your environment, then filters every retrieval by the requesting user's source permissions and data classification before anything is placed in a prompt. Answers come back with citations to the passages used.
A shared index flattens your permissions.
The quickest way to build RAG is to copy everything into one vector index and let the model search it. That quietly grants every user access to every document the index contains. The HR investigation folder, the board deck, and the patient chart all become retrievable by anyone who asks a well-phrased question.
Data.co keeps source permissions attached to every chunk at ingest, re-syncs them on a schedule, and applies them as a filter during retrieval for the identity making the request. Classification labels add a second check: a policy can exclude PHI or privileged material from a use case even where a user technically has source access.
What happens between a question and an answer.
- 1
Ingest with ACLs
Connectors pull content and its access control lists. Each chunk stores source, owner, permissions, and classification.
- 2
Embed locally
An embedding model running in your environment converts chunks to vectors; text and vectors never leave the boundary.
- 3
Filter, then rank
Retrieval applies the requester's permissions and policy before similarity ranking, so unauthorized chunks are never candidates.
- 4
Generate with citations
A private model answers from the permitted passages and returns references to each source used.
pgvector by default. Other stores only if you need them.
Vector indexes sit behind an abstraction. Most deployments never need anything beyond PostgreSQL.
| Option | When it fits | Notes |
|---|---|---|
| pgvector (default) | Most deployments, including large document sets | Runs in PostgreSQL you already know how to back up, secure, and monitor |
| External vector store | You already operate one, or have specialized scale needs | Optional, behind the same retrieval interface; never required |
| Hybrid search | Exact terms matter: codes, part numbers, citations | Keyword and vector results combined before permission filtering |
The caller's identity is part of every query.
Applications pass the end user's identity through the gateway. Retrieval filters on it automatically; there is no unfiltered mode for application tokens.
{
"collection": "policies-and-procedures",
"query": "What is the retention period for claim appeals?",
"top_k": 8,
"on_behalf_of": "user:[email protected]",
"purpose": "member-services"
}Illustrative example · final API surface documented in /docs
Built to be checked, not just trusted.
- Citations returned with every answer
- Permissions re-synced from source systems on a schedule
- Deleted source documents removed from the index
- Classification labels applied at ingest
- Retrieval decisions written to the audit log
- Evaluation sets to test retrieval quality over time
Most answer quality is decided before the first question.
Parsing determines what the model can ever see. Tables flattened into run-on text, headers and footers repeated on every page, and scanned PDFs without OCR all produce chunks that rank poorly or mislead. It is worth reviewing parsed output for each major document type before indexing a whole repository.
Chunking is a trade-off. Small chunks rank precisely but lose surrounding context; large chunks keep context but dilute similarity and consume prompt budget. Structure-aware chunking, which splits on headings, sections, or record boundaries and carries the document title and section path into each chunk, usually beats fixed-size windows. Overlap between chunks helps with passages that straddle a boundary, at the cost of a larger index.
What drives storage and memory for the index.
Considerations for planning PostgreSQL capacity. Measure on a representative sample before committing to a full ingest.
| Factor | Effect | Planning note |
|---|---|---|
| Number of chunks | Rows in the vector table | Driven by corpus size, chunk size, and overlap |
| Embedding dimensions | Bytes per vector | At 32-bit floats, roughly 4 bytes per dimension per chunk, before index overhead |
| Index type | HNSW or IVFFlat in pgvector | HNSW generally favors query speed and recall at the cost of build time and memory |
| Metadata and ACLs | Extra columns per chunk | Permission filters need indexed columns to stay fast |
| Re-embedding | Full rebuild of vectors | Plan GPU time and a parallel index when the embedding model changes |
Keeping an index correct after launch.
- 1
Schedule sync
Set content and permission sync intervals per source, shorter for systems where access changes often, such as matter or case files.
- 2
Propagate deletions
Confirm that removed source documents disappear from the index; deletion lag is a retention question as well as a quality one.
- 3
Back up like a database
The index is PostgreSQL. Include it in existing backup and restore testing, along with the ingest state needed to resume sync.
- 4
Re-run evaluations
Keep a set of real questions with known source passages and re-run it after content, chunking, or model changes to catch regressions.
Questions buyers ask
How are permissions kept current when they change in the source system?
Connectors re-sync access control lists on a schedule you set, and some sources support change notifications. Retrieval always evaluates the stored permissions at query time, so a revoked user stops receiving those chunks after the next sync.
Do we need a dedicated vector database?
No. pgvector in PostgreSQL is the default and handles most deployments. External vector stores are supported behind an abstraction if you already run one, but they are never required.
Can RAG reduce hallucinations?
Grounding answers in retrieved passages and requiring citations makes unsupported answers easier to spot and less frequent, but it does not eliminate them. Citations let users verify, and evaluation sets let you measure retrieval quality as content changes.
Which file types and systems can be indexed?
Common office documents, PDFs, HTML, and text, plus structured records from databases and business applications through connectors. Scanned documents need an OCR step, which also runs locally.
Can different departments have separate collections?
Yes. Collections can be scoped by department, matter, deal, or any other boundary, and policies control which identities and use cases can query each one.
What about instructions hidden inside indexed documents?
A retrieved document can contain text written to steer the model. Retrieval cannot reliably detect every case, so the defenses sit around it: the model answers only with what the user could already see, citations expose the source, and any tool use is governed separately at the MCP gateway.
How do we measure retrieval quality?
Build a set of representative questions labeled with the passages that should answer them, then track how often those passages appear in the top results. Re-run it after changes to chunking, embedding models, or content to catch regressions early.
What happens when we change the embedding model?
Vectors from different embedding models are not comparable, so the collection must be re-embedded. A common approach is to build the new index alongside the old one, evaluate it, and switch over when it passes.
See it against your own use case.
A 30-minute technical walkthrough, policy simulation on your scenarios, and a deployment recommendation for on-prem, private cloud, or hybrid.