How to prepare company documents for private AI search

Point AI search at a file server as it is and it will quote outdated procedures, miss every scanned contract and show people files they were never meant to see. So first decide which sources go in and who may see each one, then clean them: remove duplicates and outdated versions, turn scans into text, and attach metadata such as owner, date and access group. Index the result on infrastructure you control, with the same permissions as the source systems, and test search on real questions before anyone relies on it.

Why preparation decides the result

Private AI search, often built as RAG (retrieval-augmented generation), finds relevant passages in your documents and lets a language model answer from them. The answer is only as good as what the search finds. If the index holds three versions of a procedure, the answer may come from the old one. If a scanned contract was never turned into text, the system cannot see it at all.

Permissions matter just as much. Microsoft, for example, states that Copilot “only surfaces organizational data to which individual users have at least view permissions” and asks customers to get their permission models right. A private system you build yourself works the same way: it will show people whatever your file shares already overshare. OWASP lists sensitive information disclosure and vector and embedding weaknesses, including leaks between user groups that share one index, among the top risks for LLM applications.

Step by step

1. Start with one team and one kind of question. For example, “how do we handle a warranty claim for product line X”. List where the answers live: a SharePoint site or network share, the ERP, a ticket system, shared mailboxes. Leave everything else out of the first round.
2. Map owners and permissions. For each source, write down who owns it and which groups can read it today. Carry those groups into the index, and filter results by the asking user’s groups before the model sees anything. If you find folders open to everyone that should not be, fix them at the source.
3. Leave out what should not be searched. HR files, salary sheets, personal data the use case does not need, drafts nobody approved. The GDPR data minimisation principle (Article 5(1)(c)) limits processing to what is necessary for the purpose. Less data in the index also means fewer wrong answers.
4. Deduplicate and mark the valid version. Group near-identical files, keep the current approved version, and exclude or label the rest. When you see a file named “procedure_final_v3_NEW.docx”, you know the documents need a status field.
5. Turn everything into clean text. Born-digital PDFs and Office files can be parsed directly with tools such as Apache Tika or Docling; Docling also keeps tables and headings. Scans need OCR first, for example Tesseract or OCRmyPDF. Check a sample by hand, because tables, stamps and handwritten notes are where extraction usually breaks.
6. Add metadata. At minimum: source, owner, date, version or status, language and access group. Metadata lets you filter (“current procedures only”), show the source under each answer and remove a document from the index when it is withdrawn.
7. Split documents along their structure. Cut at headings and sections instead of a fixed character count, keep tables whole, and store the document title and section name with each chunk so every answer can point to its source.
8. Keep the index inside your network. Embeddings and the model can run on your own servers or in a private cloud. The vector index can live in PostgreSQL with pgvector, and models can be served with tools such as Ollama or vLLM. Treat embeddings as sensitive data: OWASP notes that embedding inversion attacks can recover significant amounts of source information.
9. Test on real questions. Collect 30-50 questions from the people who will use the system, write down the correct answer and the document it comes from, and check whether search finds that document. Run the same test after every change to the pipeline.
10. Plan the updates. Documents change every week. Decide how new files get in, how withdrawn ones get out, and who reruns the test questions each month.

An illustrative example

Picture a service company where field technicians call the office to ask about repair procedures. The answers sit on a network share with years of manuals, many of them scanned, and in a ticket system with past fixes.

A sensible first round covers only the manuals for the two most common device families and closed tickets from the last two years. The team removes duplicate manuals, runs OCR on the scans, tags each file with device family, version and status, and keeps tickets readable only by the service group. The test set is the 40 questions the office hears most often. The system goes to technicians only after it finds the right document for most of them, with a link to the source under every answer.

Common mistakes

• Indexing the whole file server on day one. Every old version and every overshared folder ends up in the index, and nobody can trust the answers.
• One service account that reads everything. Search then ignores who is asking. Filter by the user’s groups.
• Skipping OCR checks. A scan that turns into garbage text is invisible to search, and nobody notices until an answer is missing.
• Answers without sources. People need to open the document to verify the answer and to trust the next one.
• Treating it as a one-off import. Without an update process the index drifts away from your real documents within months.

Next step

Choose one team and one recurring question, list the sources that answer it, and check who can read each of them today. That list is the scope of your first index. Once search works, the same documents can feed an AI agent that acts on them, and our guide on how to choose the first process for an AI agent helps with that decision.

Rebels Software builds private AI that runs on your infrastructure, as part of the custom platforms and applications we deliver. In a Discovery workshop we map your sources and permissions with you, then prove one case as a PoC on your real documents, inside your network. After deployment, Rebels Care keeps the pipeline and the index up to date. If that matches where you are, book a call on this page or start from the Rebels Software homepage.