Why Connecting AI to Your Company Documents Isn't Enough

Imagine an employee asking an AI assistant a simple question:

"What is our current procedure for approving a new client?"

The assistant searches the organization's documents and returns a clear, professional answer. It even cites a real policy document.

There is one problem. The policy it cited was replaced six months ago.

The answer sounds right, and the citation is genuine. It is still wrong. This is one way AI can fail inside organizations, and the cause is often not the model. It is the knowledge behind it.

Organizations are moving beyond general-purpose AI

AI is now part of everyday work, but turning individual gains into organizational results is proving harder.

In McKinsey's State of AI survey, published in August 2026 with 1,719 respondents across 97 countries, 80% said AI had improved their individual productivity. Only 37% reported a positive impact on EBIT at the organization level, a figure essentially unchanged from 2025.

These are self-reported figures, and they don't explain why the gap exists. But they make an important distinction clear: using AI and getting measurable organizational value from AI are not the same thing.

The same survey found that knowledge management is among the functions where respondents most often report scaling AI agents. Organizations are connecting AI to what they already know: policies, manuals, contracts, client files, research and procedures.

Connecting those documents is only the beginning.

How AI uses organizational knowledge

One widely used approach is retrieval-augmented generation (RAG). Instead of expecting a language model to already know your internal information, the system looks it up when someone asks a question:

1. Retrieve: search approved sources for relevant content.
2. Provide context: pass that content to the language model.
3. Generate: produce an answer based on it, ideally with references to the original sources.

An engineering team could ask "What maintenance procedure applies to this equipment model?" and get a summarized answer that points to the relevant manual, instead of searching through several of them.

RAG lets answers reflect current documents without retraining the model every time something changes. It does not, on its own, make those answers reliable. Four factors are particularly important.

1. Finding the right information

Some of the most useful evidence here comes from a team that built one of these systems and reported what they learned.

IBM researchers describe building an AI assistant that answers questions from IBM's own product documentation. Their paper reports two kinds of evidence, and both are instructive.

A controlled experiment: small content changes, large effects. In a separate benchmark experiment using 189 questions, the model reported one figure as 180 ppm instead of 280 ppm until the source text was edited to state the value clearly. In that experiment, minor edits like this raised the success rate to 100%. That result describes the controlled test, not the production system.

Production experience: missing content and failed search are different problems. When the team reviewed real user questions to the production assistant, they found that in July, no documentation topic existed for 40% of valid questions, so writers were brought in to fill the gaps. Separately, by December, search failed to find the relevant topic 47% of the time, even though the right content now existed for most valid questions.

The practical lesson is that document quality can matter as much as model choice. The IBM authors offer five formal content guidelines that apply well beyond software documentation:

1. Simplify complex tables. Tables with merged cells or missing column headings are hard for language models to read.
2. Explain graphics in text.
3. Add a summary to long tutorials and procedures.
4. Introduce every list with a clear lead-in sentence.
5. Avoid multi-level nesting.

More broadly, the paper recommends writing content to answer the questions users are likely to ask, collecting those questions from real users, and keeping each topic short, complete, accurate and up to date.

When an answer is wrong, it also helps to ask which problem you have: is the information missing, did search fail to find it, or did the model misread it? Each needs a different fix.

2. Knowing which information to trust

Consider a company with three versions of the same policy: one approved last year, one draft, and the current version.

If an AI retrieves all three without knowing their status, it may give an outdated or inconsistent answer, exactly like the example at the start of this article.

Reliable knowledge systems need to know more than what a document says. They need to know:

• Status: approved, draft, or retired.
• Version and effective date: which one applies today.
• Owner: who is responsible for keeping it accurate.

Retired and draft material should be removed from search or clearly marked. And when sources are missing or contradict each other, the most useful thing an assistant can say is that it cannot establish a reliable answer, and point to who can.

Trust also includes where content comes from. The OWASP Top 10 for LLM applications describes how poisoned or unverified documents can manipulate AI answers. In one of its examples, a résumé carries hidden white-on-white text instructing the AI to recommend the candidate. Its guidance is to validate documents before they enter the knowledge base and to accept data only from trusted sources.

3. Respecting information access

Not every employee should see every document. Financial records, client files, employee information and legal advice all carry different restrictions.

Those restrictions must still apply when people use AI. An assistant should never reveal information simply because the underlying search system can reach it.

OWASP lists unauthorized access and data leakage as a leading risk for these systems, including cases where one group's content is retrieved in answer to another group's questions. Its recommended controls are concrete:

• Use permission-aware search: filter results by each user's access rights at the moment of retrieval, not just in the interface.
• Partition data that belongs to different teams, clients or tenants.
• Classify content so access levels are applied consistently.
• Keep detailed, tamper-resistant logs of what was retrieved, by whom.

A simple test is worth running before launch: ask the same question as users with different roles, and confirm that each gets only what they are allowed to see.

4. Proving that answers are reliable

A citation shows where an answer came from. It doesn't prove the answer is correct.

Researchers at Stanford tested specialized AI legal research tools on more than 200 legal queries. The tools made fewer errors than a general-purpose chatbot, but in that study each still produced incorrect information more than 17% of the time. That finding applies to the specific systems they evaluated, not to every RAG system. It does show that retrieval and citations are not guarantees.

Evaluation is also harder than it looks. The IBM team found that standard automated benchmark metrics were not useful for judging answers to new questions from real users, and adopted what they call a "human in the lead" approach. They also found that users gave feedback on less than 1% of answers, so thumbs-up buttons alone won't tell you whether a system works.

In practice, reliable evaluation looks like this:

• Build a set of real questions with known correct answers and the sources that support them.
• Check whether each answer is accurate, supported by its source, current, and properly restricted.
• Re-run the same questions whenever documents, models or settings change.
• Keep human review for consequential decisions.

Where private AI fits

All four challenges exist whether an AI system runs in the cloud or on your own infrastructure.

On-premises AI addresses a different question: who controls the environment where your organization's knowledge is processed? (We compare deployment options in more detail in On-Premises AI vs Cloud AI.)

With a well-designed private deployment, document processing, retrieval and AI inference can stay inside infrastructure you control. That gives you direct say over which models are used, who can reach the system, when it is updated, what is logged, and whether it connects to anything outside.

Canada's privacy regulators frame responsibility in a way that applies to any deployment model. The Office of the Privacy Commissioner's principles for generative AI call for safeguards that match the sensitivity of the information, reasonable steps to make AI outputs as accurate as their purpose requires, and accountability that, in their words, "rests with the organization, and not with any kind of automated system."

Private deployment gives you direct control over certain technical safeguards, such as where data is processed, who can reach the system, and what it connects to. Meeting governance and privacy obligations still requires appropriate implementation. The OWASP risks above apply on your own servers too. Reliable results still require good knowledge management and sound security engineering.

Start with one problem, not every file

The most practical first step is to choose one area where people regularly struggle to find trustworthy answers: a policy library, a set of technical manuals, or a collection of operating procedures. Then:

1. Identify the authoritative documents and who owns each one.
2. Clean them up using the content guidelines above, and retire outdated versions.
3. Apply the right access permissions and confirm they carry through to search.
4. Build a focused assistant and test it with real questions from the people who will use it.
5. Measure answer quality, usability and time saved before expanding.

A focused start makes it possible to show value early. It also tells you whether a poor answer came from missing information, weak retrieval, or the model itself.

Conclusion

The value of organizational AI doesn't come from how many documents it can reach. It comes from whether people get useful, reliable and properly authorized answers.

A powerful model can produce impressive responses, but it cannot fix an outdated policy, resolve a contradiction between documents, or invent a missing access rule. That takes reliable knowledge, effective retrieval, clear governance, and infrastructure that fits how your organization operates.

Your company's AI is only as good as the knowledge behind it.

At On Premises AI, we work on bringing practical AI to infrastructure organizations control. If you're exploring a private knowledge assistant, see how our platform approaches it, or book a conversation with our team.

References

1. McKinsey & Company. The State of AI: Global Survey 2026. August 2026.
2. Packowski, S., Halilovic, I., Schlotfeldt, J., & Smith, T. (IBM). Optimizing and Evaluating Enterprise Retrieval-Augmented Generation (RAG): A Content Design Perspective. 8th International Conference on Advances in Artificial Intelligence, 2024.
3. Magesh, V., Surani, F., Dahl, M., Suzgun, M., Manning, C. D., & Ho, D. E. Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools. Journal of Empirical Legal Studies, 2025.
4. OWASP GenAI Security Project. LLM08:2025 Vector and Embedding Weaknesses. 2025.
5. Office of the Privacy Commissioner of Canada. Principles for responsible, trustworthy and privacy-protective generative AI technologies. December 2023.