AI, Actually: How Do You Know an AI Is Actually Good? (Evals, Explained)

Key Insight

A demo is a highlight reel, not a test.

When a stakeholder watches a chatbot nail three sample questions, their brain does the same thing it does at a magic show. It fills in the gaps. It assumes the edge cases work. It assumes the tone holds up under pressure. It assumes the thing doesn't hallucinate when asked about your company's actual refund policy.

Evals — short for evaluations — are the discipline that replaces that assumption with measurement. Think of them as unit tests for a system that doesn't return the same answer twice. That sounds hard, and it is, but the alternative is shipping blind.

The dirty secret of the AI pilot graveyard is that most of those projects had perfectly capable models. They failed because nobody defined what "good" meant in testable terms before go-live.

Why Teams Miss This

Traditional software testing runs on determinism. You give a function input X, you expect output Y, and if you get Z, the test fails. Clean, binary, automatable.

LLMs break all three assumptions:

• Non-determinism. Even at temperature zero, model updates and context changes cause drift. The output you validated last Tuesday may not be what ships this Friday.
• Semantic equivalence. "The return window is 30 days" and "You have a month to return it" are the same answer. A string-match test fails one of them.
• Subjectivity. Whether a customer support response is "helpful" is a rubric call, not a boolean.

So teams either skip testing entirely ("it looked good in Slack") or they apply traditional unit tests where they don't belong and get false confidence. Neither works in production.

How to Actually Do It

Mature eval setups run at four levels. You don't need all four on day one — but you need to know which layer covers which failure mode.

Layer 1: Functional correctness — tasks with real right answers. Entity extraction, classification, JSON validation, SQL generation. These you can test exactly like traditional software, and you should run them in CI:

def test_entity_extraction():
    text = "Apple Inc. was founded by Steve Jobs in Cupertino, California."
    result = extract_entities(text)
    assert "Apple Inc." in result["organizations"]
    assert "Steve Jobs" in result["people"]

Fast, cheap, catches regressions when you swap models or tweak prompts. Start here.

Layer 2: Semantic similarity — when exact match is too strict. Summarization, paraphrasing, open-ended Q&A. Use embedding cosine similarity against reference answers and set a threshold:

score = semantic_similarity(generated_summary, reference_summary)
assert score > 0.82, f"Summary drift detected: {score:.3f}"

Tune the threshold per task. Summarization tolerates more variation than factual extraction.

Layer 3: LLM-as-judge — for subjective quality like tone, safety, and helpfulness. You use a stronger model (or a specialized evaluator) to score the output against a rubric. Yes, you're using AI to grade AI. That's fine. The key is a stable, versioned rubric so the judge doesn't drift either. Tools like Braintrust, LangSmith, and Arize Phoenix have this built in.

Layer 4: Production monitoring — continuous sampling of live traffic, scored automatically, with alerts when quality drops. This is where you catch the problems that never showed up in your golden dataset because users are more creative than your test authors.

The minimum viable eval stack for an enterprise pilot: a golden dataset of 50-100 real examples with expected outputs, a Layer 1 suite running in CI, and a Layer 3 rubric reviewed by a domain expert before launch. That's it. You can iterate from there.

What We've Learned

Before your next AI pilot review, ask yourself what would have to be true about the output for you to ship it confidently.

If the answer is "it seemed good in testing," you don't have an eval. You have a demo with extra steps.

Write down the rubric. Build three functional tests. Run them before the stakeholder review, not after. The gap between "impressive demo" and "production-ready" is almost always just that someone sat down and defined what good actually means.

That's evals — QA for a system that doesn't behave like software.

Sources

• LLM Evaluation in Production — nepexgroup.com
• LLM Evaluation Framework Benchmark 2026: DeepEval vs RAGAS vs Promptfoo — aiml.qa
• LLM Evaluation for Enterprise, Beyond Benchmarks — knowlee.ai
• Braintrust — LLM eval and tracing platform
• LangSmith — LangChain's eval and observability layer
• Arize Phoenix — open-source LLM eval and tracing