Back to blog

· by Omar Demaría

How to QA an AI app: evaluating a chatbot or RAG without exact answers

An LLM answers differently each time, so 'expected equals actual' doesn't work. A practical guide to testing chatbots and RAG: reference dataset, rubrics, thresholds and safety tests.

  • llm
  • generative ai
  • testing
  • rag

Short answer: an LLM-powered application can't be tested by comparing the response to an exact string, because the same question can yield different answers. You test it with a reference set of cases, evaluation criteria (rubrics), acceptance thresholds and continuous runs, closer to monitoring a metric than to a classic unit test.

Why traditional testing falls short

In deterministic software, input → output is predictable and an assertEquals is enough. With an LLM:

  • The same question can produce different answers between runs.
  • A model change, a version bump or a prompt edit alters behavior without touching your code.
  • "Correct" is usually a judgment (is it relevant? safe? faithful to the source?), not an equality.

A test that passed yesterday can fail today with nobody changing anything. The answer isn't to give up on testing, but to change what you measure.

The five layers of a QA strategy for AI

1. A reference dataset ("golden set")

Collect 30 to 100 real cases: typical questions, edge cases, questions the system should not answer, and examples of queries that already failed in production. Each case records what is expected (the right idea, not the literal text) and, where relevant, the source that should support it.

2. Deterministic checks first

Before using another model as a judge, validate what code can verify:

  • Does the output respect the format (valid JSON, required fields)?
  • Is it in the right language and within the expected length?
  • Does it contain data it shouldn't (emails, phone numbers, keys)?
  • Does it cite sources that actually exist in your knowledge base?

These are fast, cheap and unambiguous.

3. Rubrics for the subjective parts

For relevance, tone or accuracy, define a clear rubric (for example, 1 to 5 with a description of each level). You can apply it with sampled human review or with an LLM as a judge. If you use a model as a judge, treat it as an instrument that can also be wrong: calibrate its results against a sample scored by people.

4. In a RAG, evaluate retrieval and generation separately

If an answer is bad, the problem can be in two different places:

  • Retrieval: were the right pieces of your documentation found?
  • Generation: did the model use them faithfully or make things up?

Measuring them separately tells you where to fix. Open-source tools such as promptfoo, Ragas or DeepEval help organize this kind of evaluation.

5. Safety and adversarial cases

Test prompt injection attempts, out-of-scope requests, leakage of internal data and answers that shouldn't be given (medical or legal advice, for instance). These cases belong in the dataset from day one.

Putting it into practice

  1. Define thresholds per metric (for example, "at least 90% of cases with acceptable faithfulness") instead of demanding 100%.
  2. Run the evaluation on every prompt, model or knowledge-base change, as part of CI.
  3. Store historical results to see trends and catch regressions.
  4. Keep the dataset alive: every production failure becomes a new case.
  5. Always keep human review on high-risk decisions.

A common mistake

Testing only that "it answers something". A chatbot can answer with total confidence and be wrong. The QA question isn't "did it answer?" but "did it answer well, safely and with the right information?".

If your team has already tested AI features and wants to structure the process, you may also like generative AI in software testing: use cases and risks and how AI strengthens your QA team.

Launching an assistant or an AI feature and want an evaluation before going to production? Let's talk.