Why RAG Projects Start with Data: 7 Questions Before the First Pilot

In RAG projects, the conversation often starts very technically: Which language model should we use? How large should the text chunks be? Which database do we need?

These are valid questions. But they usually come too early. Because even the best AI cannot give a reliable answer if it works with old, contradictory, or incomplete information.

RAG can be easily explained: Imagine a very fast assistant. Before it answers, it searches the company's shared documents for relevant information. If it finds the correct and current work instruction there, it can help well. If it finds three different versions without dates, the answer becomes a gamble.

A good RAG pilot, for me, therefore begins with three simple questions: What should the system answer? Where is the correct information? And who can say if the answer is correct?

This is precisely where it often gets exciting for companies. Knowledge is rarely neatly organized in one place. Some is in SharePoint, some on the network drive, a lot is in emails, and some is only in the minds of individual employees. That's completely normal. You just need to be aware of it before you start building.

Even studies on data quality in RAG systems show: problems often arise right at the start of reading in and preparing content, and then they permeate the entire system. Optimizing the technology at the end does not solve this root cause. Here is the study.

„We want to leverage our knowledge with AI“ sounds good, but it's still too general. It would be better to say, for example: Customer service should be able to find the currently valid warranty policy within a few seconds. Or: Project managers should be able to compare requirements from multiple handbooks.

The more concrete the task, the easier it will be to check later whether the system really helps.

An honest list is worthwhile here: SharePoint, network drives, CRM, ticketing system, emails, wiki, or specialized software. After that, it will be clarified which source is binding. After all, an approved manual carries a different weight than an email that is two years old.

Important content requires a responsible person or department. Let's take an internal travel expense policy as an example: Accounting must be able to say which version is valid and which answer would be correct. Without this technical review, IT can only guess.

A filename like „Richtlinie_neu_final_v2“ doesn't help humans or AI much. Documents need at least a date, a status, and clear assignment. Otherwise, the system might find an old rule when a new one has long been in effect.

An internal AI system must not become a backdoor to confidential data. If employees today do not have access to personnel files or sensitive project data, this information must also not become visible through an AI response.

Duplicate files, poorly legible scans, different terms for the same thing, or huge consolidated documents are typical examples. Not everything needs to be perfectly cleaned up before the pilot. But known problems belong on the table so that expectations remain realistic.

A few spontaneous questions in a presentation are not enough. A list from everyday work is more useful: real questions, the expected answer, and the valid source for it. The system should also be able to clearly state if it cannot find a reliable answer.

“Pilot-ready” does not mean everything across the company has to be perfectly organised. It is enough to define one valuable domain clearly and learn within controlled boundaries.

1. Set task: Who should be supported with what specifically?
2. Capture data channels: Where is important, supplementary, and confidential information located?
3. Check sources: What is current, who is allowed to see it, and who is responsible for it?
4. Collect real test questions: What answers are really needed in everyday life?
5. Start small. A clearly defined area provides more insights than a huge pilot with everything simultaneously.

Also this NIST AI Risk Management Framework places responsibilities, risk assessment, and ongoing measurement at the core. Practically translated, this means: sources, rights, and quality control are part of the solution from the start, not later added to a documentation folder.

Data preparation is not the tedious prerequisite before the actual AI project. It is an essential part of it. This is where it's decided which questions can be answered, who is allowed to see the answers, and whether the system will still work reliably in a year.

A structured Data & AI Readiness Assessment This provides a clear overview: Where do we stand, what data can we use effectively, and where should we start? This way, the foundation is clarified first before unnecessarily spending money on technology.

• Data Quality Challenges in Retrieval-Augmented Generation
• NIST AI Risk Management Framework