Measuring RAG Quality: Test Questions for Product and Quotation Knowledge

“Which seal fits this version?” Suppose a member of the sales support team asks the internal knowledge base this question. The answer names a material, an application range and a data sheet. Would you use it for the next stage of quotation preparation?

Our suggestion: first check these answers against a small, fixed set of questions. Assess more than whether the wording sounds clear: compare it with the expected product information and the relevant source passages. For this purpose, we propose a TRIXNER Working Model below. It is a practical test template, not a standard or proof that a system is error-free.

RAG stands for Retrieval-Augmented Generation. The approach combines retrieving information from an external knowledge base with generating text. The research paper by Lewis and colleagues describes combining a language model with a retrievable document collection.

Microsoft’s documentation on RAG evaluators distinguishes, among other things, between evaluating document retrieval, checking whether the answer is supported by the supplied context, and assessing completeness against a reference answer. We use this distinction as a conceptual starting point, not as a recommendation for a particular vendor.

For your own test sheet, we recommend three separate questions:

• Retrieval: Was the intended source for this question found?
• Answer: Do the statements agree with the source passages and the prepared reference answer?
• Usability: Does the answer meet the requirements you set for this step in the workflow?

Record the observations separately. This helps you specify in the follow-up discussion whether search, wording or your subject-matter requirements need closer examination.

For an initial round, we suggest about 20 questions from a clearly defined product area. This number is a pragmatic starting point, not a scientific minimum. Choose questions whose expected answers a knowledgeable person can check against approved documents.

The sales support team can compile the questions and document the test round. We suggest involving engineering or product specialists in preparing reference answers and reviewing disputed results. This working model does not require an additional, multi-stage approval process.

For each question, record:

• the exact wording of the question, along with the product and variant where known;
• the expected key statements or clarification question;
• the reference document, version and specific source passage;
• the test account used and its intended access rights;
• the sources actually retrieved and the full answer;
• your assessment and the next review step.

Practical Tip: Write down the expected key statements before opening the AI answer. Instead of just “must be correct”, specify, for example, “state the material and operating limit from section X; ask for clarification if the variant is missing”.

The following questions are invented examples, not product specifications or customer cases. Replace the placeholders with your own test data. The expectations are suggestions for this test, not claims about how every RAG system behaves.

Use synthetic data and clearly separated test accounts for the access-rights test. Assess the results only for the cases tested; do not treat a successful round as a complete security assessment.

For the assessment, we recommend three simple categories: “met”, “partially met” and “not met”. Score retrieval, answer and usability separately. Also record exactly what is missing or incorrect.

For an internal evaluation sheet, we suggest listing the number of cases tested and the number that met expectations in each test group. Alongside these figures, show the individual cases you classified as critical. Define this classification before testing, for example for an invented technical approval or disclosure of protected terms. We deliberately do not prescribe a universal target rate.

For our working model, we recommend resolving these critical cases before broader use and then retesting them. Decide within your company which remaining limitations you are willing to accept for the intended use.

Practical Tip: Deliberately include a question whose answer is absent from the supplied documents. Write down beforehand which clarification question or acknowledgement of missing information would be acceptable. Evaluate this case explicitly rather than using only questions with an available answer.

If you subsequently change the documents, search or answer instructions, we recommend repeating the test with the existing questions. Record the document versions, system configuration and test date. Retain the previously defined expectations for comparison questions unless their factual basis has changed.

We also suggest adding new questions that were not used during the revisions. Have disputed assessments checked by a subject-matter expert. Treat the result as an observation from this test round, not a guarantee for every future enquiry.

Our conclusion: start with a manageable product area and clear expectations. Check which sources were found, whether the answer represents them correctly, and whether it is sufficient for the intended step in your workflow. This focuses the test on the decision you actually need to make: what should the knowledge base support, and where would you still want a clarification question?

For Further Planning: AI Knowledge Bases. We can discuss a suitable test scope based on your questions.

• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• Retrieval-Augmented Generation (RAG) Evaluators for Generative AI – Microsoft Foundry
• AI knowledge bases | TRIXNER