Build the first working version before adding complexity.

Prerequisites from zero

• Know that a large language model generates text one token at a time.
• Know what a prompt is: the instructions, examples, and context sent to the model.
• Know what a test case is: an input plus the behavior you expect.

What to build

1. Start with 3 task examples, then expand toward 12 examples with inputs, expected facts, and unacceptable mistakes.
2. Create two prompt variants so the learner can compare a baseline against a proposed change.
3. Run each example, store the model output, and score it with exact checks, rubric checks, or a model-graded judge.
4. Print a report with pass rate, failed examples, cost estimate, and a release recommendation.

Eval checks

• All starter examples run without changing the evaluation code, and the harness is ready to expand toward 10 or more cases.
• The report shows baseline score, candidate score, and every failed case.
• A candidate prompt cannot pass if it omits required facts or invents unsupported facts.

Failure modes

• The eval set is too small or too easy, so it cannot catch regressions.
• A model judge rewards fluent answers even when they are factually wrong.
• The team optimizes for the eval and forgets real user traffic.

Prerequisites from zero

• Know that retrieval-augmented generation means retrieving external evidence before generating an answer.
• Know what a chunk is: a smaller searchable piece of a document.
• Know what an embedding is: a vector representation used for semantic search.

What to build

1. Create a docs folder with three short source documents and metadata such as title, URL, and date.
2. Chunk the documents, embed each chunk, and store the vectors in a local index.
3. Build a query route that retrieves top-k chunks, assembles a grounded prompt, and returns citations.
4. Add a refusal path when the retrieved evidence is weak or missing.

Eval checks

• Questions with known answers cite the right source document.
• Out-of-scope questions return a refusal instead of an invented answer.
• Changing chunk size or top-k changes measurable retrieval recall.

Failure modes

• The right document exists but the retriever misses the needed chunk.
• The model cites retrieved text that does not actually support the answer.
• Long prompts crowd out the best evidence and make generation less faithful.

Prerequisites from zero

• Know what an application programming interface is: a structured way for software to request work.
• Know what a tool schema is: the required shape of valid tool arguments.
• Know what state is: information carried from one step of the workflow to the next.

What to build

1. Define one read-only tool, such as `search_docs`, with a narrow JSON schema.
2. Ask the model whether to answer directly or request the tool.
3. Validate every requested argument before executing the tool in application code.
4. Record a trace with user goal, tool call, arguments, observation, final answer, and stop reason.

Eval checks

• The agent calls the tool when the answer requires external information.
• Malformed or out-of-scope tool arguments are rejected before execution.
• The loop stops within a fixed step limit and records a readable trace.

Failure modes

• The model chooses the wrong tool or passes plausible but invalid arguments.
• The loop keeps calling tools because the stop condition is weak.
• A write action is added before permissions, approval, and audit logging exist.

Prerequisites from zero

• Know that supervised fine-tuning trains on example inputs and desired outputs.
• Know that a train split teaches the model and a test split measures behavior after training.
• Know why prompting, retrieval, or tool use may be better than fine-tuning for changing facts.

What to build

1. Write a dataset card that states the target behavior, user population, privacy risks, and known exclusions.
2. Start with 4 example rows, then expand toward 50 examples with input, ideal output, task category, source, and reviewer notes.
3. Split examples into train, validation, and test sets without duplicates or leaked answers.
4. Define baseline, fine-tuned candidate, regression checks, and a no-ship threshold.

Eval checks

• Every example has an input, ideal output, category, and review status.
• No duplicate or near-duplicate examples cross train/test boundaries.
• The plan includes baseline comparison, safety checks, and rollback criteria.

Failure modes

• Fine-tuning is used to memorize changing facts that should live in retrieval.
• Training examples teach style but the eval only checks correctness, or the reverse.
• Private or sensitive data enters the training set without a documented review.

Prerequisites from zero

• Know what a trace is: a record of steps inside one request or workflow.
• Know what a metric is: a numeric measurement tracked over time.
• Know what an evaluation gate is: a release check that blocks changes when quality drops below a threshold.

What to build

1. Instrument one AI route with request ID, model version, prompt version, latency, token usage, tool calls, retrieval IDs, and final status.
2. Create a frozen eval set that runs before release and compares the current workflow against the candidate.
3. Set pass/fail thresholds for answer quality, refusal quality, latency, cost, and unsupported claims.
4. Write a release note template that records what changed, what passed, what failed, and how to roll back.

Eval checks

• Every eval run records model, prompt, data, and tool versions.
• The gate fails when quality drops, cost exceeds budget, or unsupported answers increase.
• A developer can inspect one failed trace and identify which stage caused the failure.

Failure modes

• Logs capture final answers but not retrieval chunks, tool observations, or prompt versions.
• The gate checks average quality but misses high-risk failure slices.
• There is no rollback path after a model or prompt regression reaches users.