Useful questions.
Careful answers.

01 / EVALUATION

When a good score is the wrong answer

A model is evaluated against a dataset, but used inside a decision. Begin by describing that decision. What happens when a prediction is wrong? Which mistakes are cheap, and which are expensive?

Keep a baseline

For classification, compare against a simple majority or rules-based prediction. For forecasting, try the previous value or a seasonal average. A complicated model that does not improve on the baseline may be solving the wrong problem.

Match the split to the use case

A random split can hide leakage when the same person, document or device appears in both training and test data. Use group-based splits for repeated entities and forward-looking splits when predicting future events. Keep the final test set out of tuning decisions.

Inspect the errors

• Use precision and recall when the two types of classification error have different costs.
• Check calibration when decisions depend on predicted probabilities.
• Break results down by meaningful groups instead of reporting only a single average.
• Review individual failures, especially confident mistakes.

A useful result includes the baseline, the test procedure and examples of where the system fails. A higher score without that context is an incomplete story.

02 / DATA

The quiet work of checking the dataset

Before choosing a larger model, inspect the information it receives. Labels and features are products of a collection process. Understanding that process can explain errors that no architecture change will fix.

Look for leakage

A field created after the outcome may predict the label perfectly but be unavailable at inference time. Near-duplicate documents can make a test set look easier than real traffic. Fit preprocessing steps on the training set only, including imputation, feature selection and scaling.

Write a small data checklist

• Document where each field comes from and when it becomes available.
• Check missing values and label distribution across time and sources.
• Look for exact and near duplicates before splitting the data.
• Manually inspect a sample of disagreements and ambiguous labels.
• Preserve dataset versions so experiments can be reproduced.

Missing values are not automatically errors, and outliers are not automatically noise. Decide what they mean before removing them. A useful data check explains a pattern; it does not just delete inconvenient rows.

03 / RETRIEVAL

Start with retrieval, not a larger prompt

Retrieval-augmented generation separates finding relevant material from writing an answer. This creates two different things to evaluate: did the system find useful evidence, and did the generated answer accurately use it?

Evaluate retrieval on its own

Build a small collection of representative questions with known supporting passages. Inspect keyword search before adding embeddings. A hybrid approach can help when questions combine exact names with broader concepts, but it should earn its complexity through testing.

Make evidence inspectable

• Keep document titles, source identifiers and passage boundaries.
• Check whether chunk size removes context or fills the prompt with unrelated text.
• Distinguish an unsupported question from an unsuccessful retrieval.
• Test citations against their source passages rather than assuming they are correct.

Retrieved text is evidence, not an instruction source. Treat embedded instructions and untrusted content accordingly. Give the answerer a way to say that the available material does not support an answer. Retrieval can improve grounding, but it does not eliminate hallucinations.