RAG for a company: from document search to a useful assistant

Imagine a new support engineer. A customer asks whether a device can be returned after a firmware upgrade. The wiki has a procedure, a PDF contains an exception for an older model, and a chat message says the rule changed yesterday. Reading is not the problem. Finding the correct, current, authorized rule is.

A language model faces a similar problem. It can explain and compare, but it does not automatically know what your company changed yesterday. Being an impressive model does not grant access to the internal wiki.

RAG connects a model’s ability to reason with specific organizational knowledge. This article follows the whole path: from searching a folder to helping someone finish a real task.

What RAG means

Retrieval-Augmented Generation means generating an answer using retrieved information. The system first searches for relevant material, then supplies selected passages alongside the question. The model responds using that context.

Think of an open-book exam. Someone still has to select the right book, open the right page, and check the edition. An eloquent answer based on the wrong product manual remains wrong.

The original RAG paper combines knowledge stored in model parameters with retrieved external information. In practice, the term covers a wider family of designs. Retrieval can use vectors, keywords, a hybrid of both, or other stores. RAG is an architectural pattern, not a vector database brand.

Ordinary RAG does not retrain the model whenever a document changes. It supplies information for the current request. Updating a policy therefore calls for updating its searchable representation rather than training a model for every new paragraph.

Two different pipelines

Preparation and answering are separate processes. Drawing them separately removes the impression that the company database somehow migrates into the model’s brain.

When documents change

01SourcesManuals, files, wiki

02PreparationText, sections, version, permissions

03IndexWords + semantic vectors

When a question arrives

01Query and accessWhat is needed and permitted

02Retrieve and rankCurrent passages with provenance

03Model responseAnswer, citations, or abstention

Knowledge preparation runs when sources change. It extracts text, preserves structure and metadata, splits material into useful passages, and builds an index. Scanned pages need OCR. Tables need the relationship between headings, rows, and values to survive extraction.

Answering happens when a user asks a question. The system determines permitted sources, retrieves and ranks candidates, supplies relevant passages to the model, and checks response structure, link existence and whether the retrieved passage supports each material claim. A working link alone does not establish support; critical conclusions need a separate check. When evidence is insufficient, a clarification or abstention may be the correct result.

The index is a searchable representation of the sources. It must not become an ownerless second version of company truth. Every passage needs provenance and an update history.

Chunks: meaning before a fixed character count

A chunk is a passage the system can retrieve and pass to the model. Too little text loses context; too much brings unrelated material. Choose chunking through experiments on actual questions, not a lucky number from a tutorial.

For a procedure, preserve the rule and its exceptions. For an API, keep the operation, parameters, and constraints together. For an architecture decision, preserve the reason and consequences. Include the document title, product, version, and source link.

Suppose one paragraph says returns are allowed within 30 days and the next gives a different procedure for activated devices. Losing the second paragraph can produce technically successful retrieval and a dangerously incomplete answer.

Embeddings, hybrid retrieval, and reranking

An embedding is a numerical representation of text. An embedding model maps the question and passages into vectors; retrieval finds nearby representations. This can connect “restore access” to “password reset” without exact wording.

Similarity is not truth. Old and current procedures may look almost identical. Contract numbers, product codes, and error identifiers often need exact matching. Hybrid retrieval combines lexical search, such as BM25, with semantic retrieval.

A reranker can then compare the question with each candidate more carefully and change their order. It adds latency and cost. Keep it only if evaluation shows a useful improvement.

My RAG assistant combines BM25 and semantic rankings with RRF. The method uses positions in ranked lists; Elasticsearch’s documentation explains it. That reference describes a method, not a claim that my service runs on Elasticsearch.

Has RAG died?

Longer context windows and agents that select their own search tools have inspired confident obituaries. Meanwhile, employees still need the current procedure.

For a few documents and a one-off task, supplying complete documents may be enough. A SQL query, conventional search, or API call can also be the right answer. An extra RAG service is not obligatory architecture decoration.

But a large context window does not automatically solve selection, freshness, access control, or repeated input cost. An agent that searches, opens documents, and answers from them still performs retrieval. The orchestration has changed.

Approach	Consider it for	It does not automatically solve
Whole documents in context	A few sources, one-off analysis	Updates, permissions, large-request cost
RAG	Many changing sources and repeated questions	Source quality and support for every conclusion
SQL / API	Current stock, order status, calculations	Rules in unstructured documents
Fine-tuning	Stable behavior, style, or output format	Yesterday’s changes and revocation of document access

These approaches can work together. An assistant retrieves a policy, queries an order API, and explains the next step. The outcome matters more than membership in a fashionable architectural camp.

Start with a working conversation

Before selecting a vector store, I would collect ten to fifteen real support requests and ask:

Which questions consume the most time? Bring their actual wording, not just “we need intelligent search.”

How do employees answer today? Do they search the wiki, call a senior colleague, or reconcile PDFs? This reveals the baseline cost.

Who owns the current rule? A model cannot settle an organizational dispute between contradictory procedures.

What should the person do after reading the answer? Explain a condition, create a ticket, assign an owner? That determines the interface and integrations.

Which error is unacceptable? An outdated office address and disclosure of another customer’s contract have different consequences.

These questions produce an implementation plan because they connect technology to a person, a decision, and an accountable source owner.

A small-company implementation

Consider an illustrative service business with twenty employees, manuals, and an equipment catalogue. This is a design example, not an account of my client work.

Start with one approved collection and an internal assistant. A connector imports changes. An index preserves text, versions, and links. An API retrieves passages and calls an approved model. The interface shows the answer, evidence, and a way to report an unhelpful response.

A database supporting lexical and vector search, or a managed service, may be enough. Choose according to the existing stack, skills, and data requirements. Kubernetes is not an entrance exam for RAG.

The crucial assignments are a knowledge owner and someone who investigates failed answers. Without them, the pilot becomes a museum of old procedures. Scope the first release around a useful workflow, such as equipment troubleshooting, rather than every file in the company.

Define the allowed sources, external-provider restrictions, update schedule, and escalation route. Small company size does not mean every employee may read every contract or personal record.

An enterprise implementation

The challenge is not simply more documents. It includes multiple owners, access systems, product versions, languages, organizational boundaries, and audit requirements.

Area	Small pilot	Enterprise rollout
Sources	One approved collection	Wiki, DMS, and repository connectors; source owners and freshness targets
Access	Explicit user group	SSO, document and passage permissions, revocation, organizational isolation
Updates	Controlled batch imports	Incremental indexing, deletions, retries, lag monitoring
Quality	Questions for one workflow	Evaluation sets by product, language, and role
Operations	One accountable owner	Observability, quotas, resilience, cross-team responsibilities

Enforce permissions before material enters the model context. Information can leak through paraphrases, document titles, caches, conversation history, or logs. Test what happens when access is revoked after an earlier conversation.

Separate indexing from query serving so large imports do not block responses. Changing an embedding model may require rebuilding an index. Version it, evaluate it, and preserve a rollback path.

Questions about relationships may benefit from a knowledge graph. GraphRAG is not an obligatory maturity level. If the problem is broken table extraction, adding a graph merely connects bad data more elegantly.

Published implementations

Ring: regional support knowledge. The Ring and AWS account published on 30 March 2026 describes a validated knowledge source and a Bedrock-based RAG assistant. My takeaway is to establish source ownership and regional context before scaling conversations. This is a participant-authored case study, not my independent effectiveness audit.

PDI Technologies: scattered enterprise knowledge. The 22 January 2026 engineering account covers sources including websites, Confluence, and SharePoint, with explicit attention to documents and images. It makes visible the ingestion work hidden by the simple label “chatbot.”

My own work: knowledge across projects. LAF RAG Assistant connects documentation, code, and decision history. People and agents retrieve passages through API/MCP, with answer generation when needed. It is my working tool. A company rollout requires source and access adaptation; connecting a folder is not a complete implementation plan.

When search becomes an assistant

Users do not wake up wanting retrieval. They want to answer a customer, prepare a change, diagnose a failure, or complete a request.

Example: support request

01ClarifyIdentify product and version

02Find evidenceShow the relevant instruction

03PrepareFill a ticket draft

04ConfirmThe person authorizes submission

In the equipment example, a weak system returns five passages. A useful assistant clarifies the device model, finds the current rule, highlights the exception, and prepares a ticket for the right team. The person sees the basis and approves the action.

That requires conversational continuity, clarifying questions, citations next to conclusions, editable assumptions, and human escalation. “There is not enough evidence” can be more useful than a long answer that refuses to admit uncertainty.

Actions are a separate layer. Text in a retrieved document does not gain permission to send an email or change a record. Tools need limited authority, confirmation where appropriate, and a check of the actual operation result.

Evaluate search, answers, and work outcomes

I assess retrieval, the answer and the work outcome separately. Otherwise, fluent language can disguise poor retrieval.

Check	Question
Retrieval	Is the expected source among the first results?
Context completeness	Are exceptions, versions, and conditions preserved?
Grounding	Does evidence support every material claim?
Abstention	Does the system avoid inventing missing answers?
Access	Are restricted passages absent from responses and history?
Usefulness	Did the person finish the task faster without an incorrect action?

Include ordinary questions, missing answers, conflicting versions, exact identifiers, unauthorized requests and instructions embedded in retrieved text that try to redirect the assistant. A document supplies data; it does not define access policy or authorise tool use. Keep a separate evaluation set for changes. Repeated demonstrations using only familiar successful questions are rehearsals, not evaluation.

Count ingestion, OCR, embeddings, storage, reranking, generation, review, and knowledge maintenance. Token charges are one line of the budget. Measure time to a verified answer and the work that still required manual help.

What to do tomorrow

Choose one workflow with repeated information requests. Collect actual questions and their source documents. Assign a knowledge owner. Define a useful result and unacceptable errors. Establish conventional search as a baseline, then add generation and compare.

A pilot should produce a decision: what to deploy, who maintains it, what a useful outcome costs, and when the assistant must stop.

To apply this to your knowledge base, let’s discuss a scenario and demonstration. Start with one process and an evidence-based rollout plan rather than a technology purchase in the hope that usefulness will follow.

Yours, DPUPP