Private RAG is retrieval-augmented generation sold with a promise about where your data goes. The retrieval-augmented part is a plain enough arrangement: the system finds material relevant to your question and hands it to a model as context, which usually keeps the answer closer to your documents, though it doesn’t guarantee it picked the right passages or that the answer stuck to them.
The private part is the promise, and it isn’t a standard anyone certifies against. Our private RAG page sets out the architecture and the boundary questions in full; this post is about how a buyer tests the promise. One vendor uses it to mean tenant isolation, another means a dedicated cloud environment, a third means infrastructure you manage yourself, and a fourth means a clause about training data. Ask where storage, inference, logging, support and deletion actually happen, because the label won’t tell you.
Where the private RAG boundary actually sits
Your content comes to rest in more places than a diagram usually shows: the original files, the extracted text, the search index, the questions people ask, the answers, the logs and the backups. For each of those, ask which parties can reach it, under what terms, and for how long.
Then ask whether you can check the answers. A product that links a claim to the passage behind it gives a reviewer something to inspect, but the link itself proves nothing about whether that passage supports the claim, so correctness and coverage both still have to be tested.
And ask what it does when things go wrong: when the evidence isn’t there, when two documents contradict each other, when the only relevant document is out of date, or when the answer sits outside a user’s permission scope. Run those cases yourself rather than accepting a general assurance that the system is grounded.
What to evaluate when someone sells you private RAG
Ask how one organisation’s data is kept separate from another’s at every level: the application, the database, the search index and the file storage. Then ask how that boundary is tested, because the first answer is what they meant to build and the second is proof it works.
On citations, click one and see what opens, then ask what happens to files where OCR or parsing can’t preserve reliable coordinates. Those are the documents your awkward questions will land on.
Test access control against the workflow rather than the org chart. The user’s role, the documents in scope and the permissions in the source system all have to agree, and a role label on a settings screen doesn’t prove it works.
Bring your real documents to this, including the scans, the tables, the long appendices and the legacy formats the workflow still depends on. Record what fails to ingest, and how good the passage locations are on the files that succeed.
Then map where each part actually runs for the deployment you are buying: where files are stored, where the model runs, where the monitoring sits, where support works from and where backups live, under a managed, customer-cloud or self-hosted deployment.
Where the knowledge network comes in
Retrieval and an explicit knowledge structure solve different problems, and it’s worth knowing which one you’re buying. Marella AI maps the entities, concepts and relationships in your documents, alongside ordinary search, which can make a large document set easier to navigate. Extraction quality, and what happens to the structure as the documents change underneath it, still need testing on your own material.
When you’re buying, the difference is that search finds material, RAG hands retrieved material to a model, and a knowledge network holds selected entities and the relationships between them. Work out which of those the workflow actually needs, then check each boundary on its own.
