Skip to content

AI Engineer

Build products on top of LLMs: APIs, RAG, agents, evals and running it all in production

This map is for developers who want to ship AI features that actually work inside a product, not to do research or train models from scratch. It goes from how LLMs work at a practical level, through model APIs, prompts and context, RAG, agents and evals, to security, operations and governance. It fits best if you are already comfortable with web or backend work and want to move into AI engineering.

Start at stage 01
stages
10stages
topics
58topics
weeks, part-time
~31weeks, part-time
  • Foundation
  • Core
  • Advanced
  • TypeScriptTypeScript
  • VercelAI SDK
  • Model Context ProtocolMCP
  • pgvector
  • OpenTelemetry
  • Langfuse
  • promptfoo
  • OllamaOllama

My take

I'm a frontend developer, not an AI specialist or an ML researcher. Over the last few years I've integrated LLM features into web apps and side projects: streaming chat, structured extraction, a bit of retrieval, and written about it on this blog. This map is the path I would follow if I started as an AI engineer today, pieced together from shipping those features, reading official docs and following people who do this full time. The focus is building products on top of existing models, not training them.

What I wish I had known earlier

  • What I wish I'd known earlier is to build an eval set before tweaking prompts. Without a fixed test set I was just changing words and hoping.
  • Most of the hard work is ordinary software engineering: schemas, timeouts, retries, logging, permissions. The model call is the small part.
  • When I started I reached for RAG and agent frameworks too early, when a long prompt with prompt caching or a simple chain of steps would have solved the problem.
  • Treat every model output and every retrieved document as untrusted input, exactly the way you treat what users type into a form.
0 of 10 stages passed

Foundation01–03

  1. 01Stage 01, 3 weeksHow LLMs work, practicallyExplain tokens, context windows, sampling and embeddings well enough to predict the cost, latency and failure modes of an AI feature.
    The first time I called an LLM API I treated it like a deterministic function and was surprised when the same input gave different outputs. Understanding next-token prediction and sampling cleared up most of that confusion.

    What to learn

    Tokens and tokenizers
    Models read BPE-style tokens, not characters or words, and Vietnamese text usually costs noticeably more tokens than the same content in English. Price, context limits and latency are all counted in tokens, so measure with the provider's token counting endpoint or a tokenizer playground instead of guessing from word counts.
    Transformers and next-token prediction
    At inference time the model computes a probability distribution over the next token, picks one, appends it and repeats; attention lets each token look at the ones before it. That is why latency grows with output length and why time to first token and tokens per second are separate metrics.
    The context window and its real limits
    The window holds the system prompt, conversation history, retrieved documents, tool definitions and the output, all together. A long window does not mean equal attention everywhere: quality drops as context grows and when information is buried in the middle (context rot), so more context is not automatically better.
    Sampling: temperature, top_p, stop sequences
    Low temperature suits extraction and classification, higher values suit brainstorming or creative writing. Even temperature 0 on hosted APIs is not guaranteed to be deterministic because of batching and floating point effects, and some reasoning models restrict or ignore these parameters.
    Embeddings and similarity
    Embedding models turn text into vectors where cosine similarity approximates closeness in meaning. Know the dimension, whether vectors are normalized, and that vectors from two different models are not comparable, so switching models means re-embedding everything.
    Training stages and reasoning models
    Pretraining gives knowledge, instruction tuning and preference tuning shape behavior, and reasoning models spend extra thinking tokens before answering. Knowing this chain explains knowledge cutoffs, hallucination, and why a thinking budget trades latency and cost for accuracy.

    Do

    • Measure token counts on real data before estimating the cost of a feature.
    • Read the model card, pricing page and rate limits before choosing a model.
    • Record time to first token and total duration for every call from the very first prototype.
    • Run the same input several times and compare outputs to get a feel for variance.

    Watch out for

    • Assuming the model knows current information when it only knows up to its knowledge cutoff.
    • Treating the context window as free memory without accounting for quality loss on long contexts.
    • Believing that temperature 0 guarantees identical output every time.

    Checkpoint project: Token and cost explorer

    Build a small page that takes Vietnamese and English text, shows token counts through a provider API, estimates input and output cost for 2-3 models, and computes cosine similarity between two sentences through an embedding API. It passes when the numbers match the usage fields the API returns and the page explains why the same content costs more tokens in Vietnamese.

    • LLM
    • tokens
    • embeddings

    Quick check

    0/4 answered · 0 correct

    Get at least 3 of 4 right to mark this stage as passed.

    1. Question 1 of 4StarterYou estimate the cost of a Vietnamese chatbot by counting words and applying an English tokens-per-word ratio. What goes wrong?
    2. Question 2 of 4StarterWhy are time to first token and tokens per second tracked as separate latency metrics?
    3. Question 3 of 4StarterA test asserts the exact text a hosted model returns at temperature 0, and it fails now and then. Why?
    4. Question 4 of 4StarterYou switch to a newer embedding model but only embed new documents with it, keeping old vectors in the index. What breaks?
  2. 02Stage 02, 3 weeksWorking with model APIsCall model APIs from the server with streaming, structured outputs and tool calling while keeping cost and latency under control.
    My first prototype parsed JSON out of free text with a regex and broke constantly. Switching to structured outputs with a clear schema was the most valuable change at this stage.

    What to learn

    Messages APIs and streaming
    The major providers all use a list of role-based messages and return server-sent events when streaming is on. Learn the event types (text deltas, tool calls, usage, stop reason) so you handle streams that are cut off mid-way or stop because they hit max tokens.
    Structured outputs
    Use JSON Schema, often generated from Zod, so the provider enforces the shape through constrained decoding instead of just asking in the prompt. Still validate on the server, because the model can refuse, get truncated by max tokens, or return values that have the right type but the wrong meaning.
    Tool calling
    The model does not run tools itself: it returns a tool name and arguments, your code executes it and sends the result back in the next message. Understand parallel tool calls, how to return tool errors to the model as observations, and that tool definitions cost tokens on every request.
    Prompt caching
    Put the stable parts (system prompt, documents, tool definitions) first and the changing parts last so the prefix can be cached, which cuts input cost and latency significantly. Some providers cache automatically and others need explicit breakpoints, so check the cached token fields in the usage data to see whether the cache actually hits.
    Cost and latency budgets
    Output tokens are usually more expensive and slower than input, so capping max output is the most direct saving. Use batch APIs for offline jobs to get discounted pricing, set timeouts, retry with exponential backoff on 429 and overload errors, and always track cost per request and per user.
    Choosing models: hosted, smaller tiers or open-weight
    Hosted frontier models give the best quality, smaller tiers from the same vendor are much cheaper and faster for narrow tasks, and open-weight models give you control over data and cost at scale. A provider abstraction such as the AI SDK or an AI gateway lets you swap models without rewriting code, and the choice should rest on your own evals.

    Do

    • Keep API keys on the server and never call model APIs directly from the browser.
    • Define each schema in one place and share it between structured outputs and validation.
    • Set explicit timeouts and max output tokens on every call.
    • Log usage fields (input, output, cached tokens) and latency for every request.
    • Wrap model calls behind an abstraction so switching providers is a config change.

    Watch out for

    • Parsing JSON out of free text with a regex instead of using structured outputs.
    • Using the largest model for every call, including simple classification.
    • Putting a timestamp or other changing data at the top of the prompt so the prompt cache never hits.

    Checkpoint project: Typed invoice extractor

    Write a Next.js route handler that takes invoice text, returns typed JSON via structured outputs, streams a short summary and logs tokens, cost and latency. It passes when output that fails the schema is rejected and retried once, switching providers takes a one-line config change, and you can report p95 latency and average cost per request.

    • API
    • streaming
    • tool calling

    Quick check

    0/4 answered · 0 correct

    Get at least 3 of 4 right to mark this stage as passed.

    1. Question 1 of 4StarterYou use structured outputs with a JSON Schema so the provider enforces the shape. Why still validate on the server?
    2. Question 2 of 4StarterWhen the model calls a tool you defined and implemented in your own backend, what actually happens?
    3. Question 3 of 4StarterThe system prompt starts with Current time: 2026-11-02T10:31:07Z, followed by 8,000 tokens of fixed instructions. Cached tokens are always 0. What fixes it?
    4. Question 4 of 4StarterModel calls fail intermittently with 429 and overloaded errors. Which handling is right?
  3. 03Stage 03, 2 weeksPrompt and context engineeringDesign prompts and a strategy for feeding context to the model that is structured, versioned and measurable.
    I used to write prompts on the theory that longer is safer, full of capital letters and prohibitions. It turned out that short prompts with a clear goal, good examples and selective context gave far more stable results.

    What to learn

    Structure of a good prompt
    Separate role, task, constraints, output format and input data, usually with XML tags or headings. Keeping data apart from instructions helps the model understand the task and is a first step against prompt injection.
    Few-shot examples
    Examples are the strongest way to specify format and tone, but the model will also copy unintended patterns in them. Pick examples that are varied, close to real data and include edge cases.
    Context engineering
    Every call is a decision about what goes into the window: instructions, retrieved documents, memory, tool results. The main techniques are summarizing or compacting long histories, loading documents just in time instead of all upfront, and dropping stale tool results that no longer matter.
    Prompting reasoning models
    Reasoning models need less step-by-step coaching but clearer goals, success criteria and constraints. Tune the thinking budget to the difficulty of the task instead of maxing it out on every request.
    Prompts are code
    Keep prompts in the repo or a prompt registry with versions, review diffs like code, and tie every change to eval results. Template the variable parts so the fixed parts stay identical, which keeps them readable and cache friendly.

    Do

    • Write success criteria for the task before writing the prompt.
    • Wrap user data and documents in their own tags, separate from instructions.
    • Version prompts and record the version in the log of every request.
    • Put only what the model needs for the current turn into the context.

    Watch out for

    • Writing contradictory instructions in a long system prompt and then blaming the model.
    • Editing a prompt based on one or two bad examples without rerunning a shared test set.

    Checkpoint project: Support reply assistant

    Build an assistant that drafts replies to support tickets with three prompt versions, run against the same 20 fixed inputs in both Vietnamese and English. It passes when the prompts live in the repo with versions, there is a side-by-side comparison of the three versions, and the reason for picking the final one is written down.

    • prompting
    • context

    Quick check

    0/4 answered · 0 correct

    Get at least 3 of 4 right to mark this stage as passed.

    1. Question 1 of 4StarterWhy wrap user-provided documents in their own tags, separate from the instructions?
    2. Question 2 of 4StarterAll three few-shot examples in a review summarization prompt end with 'Overall, this is positive.' What is likely to happen?
    3. Question 3 of 4StarterA PM forwards two bad answers and asks for a quick prompt tweak. What is the right process?
    4. Question 4 of 4StarterA long agent session gets worse as old tool results pile up in the context. Which context engineering move helps most?

Core04–07

  1. 04Stage 04, 3 weeksAI UX in web appsBuild AI interfaces that stream, handle tool calls, let users stop, edit and give feedback, and make uncertainty visible.
    This is the part that felt most familiar because it is close to frontend. What I learned is that good AI UX is not the typing effect, it is users being able to control and verify the result.

    What to learn

    Streaming UI
    Use hooks like useChat from the AI SDK, or read the stream yourself through a ReadableStream, to render text, message parts and structured objects while they are still streaming. Handle errors mid-stream, retries, and keep conversation history stable when the user reloads.
    Generative UI and tool calls in the interface
    Render each tool call as its own component based on whether it is running, done or failed, for example a search result card instead of a long block of text. For actions with side effects, show a confirmation step right inside the chat flow.
    Making uncertainty visible
    Show citations that link to the exact passage, let users edit before applying, and provide undo. Design so the model can say it does not know instead of forcing it to always answer.
    Designing for latency
    Have a stop button that actually cancels the upstream request with an AbortController so you stop paying for tokens, show progress for long steps, and consider resumable streams for long-running tasks. Heavy jobs should run in the background and notify on completion instead of holding the user in a spinner.
    Safe and accessible rendering
    Model output is untrusted input: render markdown through a sanitizer and block dangerous HTML and links. Use aria-live carefully so screen readers announce the result instead of every token.
    Capturing feedback
    Thumbs up or down, user edits and whether a suggestion was accepted are all valuable eval signals. Attach each piece of feedback to a trace id so you can later look up the exact prompt, context and output.

    Do

    • Let users stop, edit and retry at every point in an AI flow.
    • Sanitize all model output before rendering it as HTML.
    • Attach a trace id to every message to connect feedback with server logs.
    • Label clearly in the UI when content is generated by AI.

    Watch out for

    • A stop button that only hides text in the UI while the server request keeps running and billing.
    • Rendering model markdown with dangerouslySetInnerHTML without sanitizing.
    • Designing the interface as if the model is always right, with no room for sources or edits.

    Checkpoint project: Writing assistant panel in Next.js

    Build a panel that suggests text edits with streaming, a stop button, suggestions shown as a diff with accept or reject, and feedback buttons stored with a trace id. It passes when stop cancels the upstream request (verified through usage), output is rendered through sanitized markdown, and a screen reader announces the final result.

    • Next.jsNext.js
    • VercelAI SDK
    • UX

    Quick check

    0/4 answered · 0 correct

    Get at least 3 of 4 right to mark this stage as passed.

    1. Question 1 of 4IntermediateThe chat's stop button hides the streaming text, but token usage on the server keeps climbing. What is missing?
    2. Question 2 of 4IntermediateWhy must model-generated markdown be sanitized before rendering it as HTML?
    3. Question 3 of 4IntermediateScreen reader users complain that the chat reads out every streamed token. Which fix is right?
    4. Question 4 of 4IntermediateWhy attach a trace id to every thumbs up or thumbs down in the UI?
  2. 05Stage 05, 3 weeksEvaluation: measure before you fixBuild an eval suite with data, graders and thresholds that runs in CI so every prompt or model change comes with numbers.
    I put evals right after UX because this is where I was wrong the longest: tweaking prompts by feel and only trying a few familiar inputs. Once evals exist, changing models or prompts stops being scary.

    What to learn

    Error analysis first, metrics second
    Read 50 to 100 real traces, take notes and group the failure types, then write evals for each group. Generic metrics like a helpfulness score rarely tell you what to fix.
    The eval dataset
    Mix samples from real traffic, edge cases and adversarial inputs, each with labels or clear pass criteria, and version the set like code. Add to it every time a new failure shows up in production so that failure cannot come back silently.
    Kinds of graders
    Prefer code-based graders whenever possible: exact match, schema checks, regex, running unit tests on generated code. Use LLM-as-judge for criteria that are hard to express in code, and human review for what matters most.
    LLM-as-judge pitfalls
    Judges show position bias, verbosity bias and a tendency to favor output from their own model family. Use clear binary pass or fail criteria instead of a 1 to 10 scale, and calibrate the judge against human labels by measuring agreement before trusting it.
    Regression testing in CI
    Run evals whenever the prompt, model or retrieval logic changes, using tools like promptfoo or Inspect, with a threshold that blocks the PR when the pass rate drops. Keep results over time so you see trends instead of a single run.
    Online evals
    In production, sample traces for automated graders and track user feedback, refusal rate, latency and cost. A/B test two prompt or model versions when a change has a big impact on users.

    Do

    • Start by reading real traces before choosing any metric.
    • Write concrete pass or fail criteria instead of scoring by feel.
    • Measure agreement between the LLM judge and humans before letting the judge gate a release.
    • Turn every production bug into a case in the eval set.

    Watch out for

    • Relying on public benchmarks to pick a model for your own problem.
    • Using an uncalibrated LLM judge and trusting its scores completely.
    • An eval set full of easy cases that always passes at close to 100 percent.

    Checkpoint project: Eval suite for the earlier assistant

    Build an eval suite of at least 50 labeled cases for the assistant from stage 02 or 03, with code-based graders and one LLM judge calibrated on 30 human-labeled cases. It passes when you report the judge's agreement rate and CI blocks a PR when the pass rate drops below the threshold you chose.

    • evals
    • testing
    • CI

    Quick check

    0/4 answered · 0 correct

    Get at least 3 of 4 right to mark this stage as passed.

    1. Question 1 of 4IntermediateYou are setting up evals for a new support bot. What should you do first?
    2. Question 2 of 4IntermediateBefore an LLM judge is allowed to block releases, what should you verify?
    3. Question 3 of 4IntermediateA feature extracts invoice fields into JSON. Which grader should you reach for first?
    4. Question 4 of 4IntermediateYour eval set has passed at about 99% for months while users still report problems. What does that most likely mean?
  3. 06Stage 06, 4 weeksRetrieval and RAGBuild a RAG pipeline with chunking, hybrid search, reranking and citations, and know when not to use RAG at all.
    I used to think RAG meant embedding everything and finding the nearest vectors. In practice quality is decided by chunking, metadata, combining keyword search, and measuring retrieval separately from the answer.

    What to learn

    When you do not need RAG
    If the documents are small enough to fit in context, put them straight into the prompt with prompt caching. If the question needs filtering or aggregating structured data, use SQL or a tool. RAG fits best for large, frequently changing corpora that need citations.
    Chunking and metadata
    Split along document structure (headings, paragraphs, tables) instead of fixed character counts, and weigh chunk size against overlap. Keep metadata such as source, section, updated date and access rights, and consider contextual retrieval: prepend a few sentences describing the document context to each chunk before embedding.
    Embeddings and vector stores
    If you already run PostgreSQL, pgvector with an HNSW index is usually a sensible first choice, with metadata filters in the same query. A dedicated vector database is worth it only when scale or operational needs outgrow Postgres; shortlist embedding models with MTEB and then verify on your own data, especially for Vietnamese.
    Hybrid search and reranking
    Combine full-text search (BM25 or Postgres tsvector) with vector search and merge with reciprocal rank fusion, because exact terms like error codes or names are often missed by vectors. Then rerank the top-k with a cross-encoder or a rerank API before putting results into context.
    Measuring retrieval and answers
    Measure retrieval on its own with recall@k and MRR on a question set labeled with the correct passages. Then measure answers for groundedness, correct citations and the ability to refuse when the information is not there.
    Permissions and freshness
    Filter by the user's permissions at query time, not after the chunks are already in the prompt. Have a pipeline that updates the index when documents change and a plan to re-embed everything when you switch embedding models.

    Do

    • Try a solution without RAG first and add RAG only with a clear reason.
    • Store source metadata with every chunk so citations point to the exact section.
    • Measure recall@k before and after every change to chunking, embeddings or reranking.
    • Enforce access control in the retrieval query, not in the prompt.

    Watch out for

    • Using vector search only and missing queries that rely on exact keywords.
    • Evaluating only the final answer, so you cannot tell whether failures come from retrieval or generation.
    • Stuffing too many chunks into context, which adds noise and cost.

    Checkpoint project: Q&A over this blog's posts

    Build Q&A over the blog's bilingual MDX posts using pgvector, Postgres full-text search, reciprocal rank fusion and reranking, with citations linking to sections. It passes when a 40-question set reports recall@5 for vector-only, hybrid and hybrid plus rerank, answers cite their sources, and the system refuses when no relevant passage is found.

    • RAG
    • pgvector
    • search

    Quick check

    0/4 answered · 0 correct

    Get at least 3 of 4 right to mark this stage as passed.

    1. Question 1 of 4IntermediateYour product docs total about 30 pages and change monthly. Users ask questions about them. What should you try first?
    2. Question 2 of 4IntermediateUsers searching for error code E4021 get unrelated passages from vector search. What is the cause and fix?
    3. Question 3 of 4IntermediateAnswers got worse after a chunking change. How do you tell whether retrieval or generation is at fault?
    4. Question 4 of 4IntermediateWhere should document access control be enforced in a RAG system?
  4. 07Stage 07, 4 weeksAgents, tool use and MCPDesign workflows and agents with clear limits, write an MCP server, and keep a human in the loop for the steps that matter.
    Agents sound exciting, so I wanted to turn everything into one. The lesson was to start with the simplest workflow that works and only let the model decide when that flexibility is really needed.

    What to learn

    Workflows first, agents second
    Patterns like prompt chaining, routing, parallelization, orchestrator-workers and evaluator-optimizer solve most problems with more reliability. A self-planning agent is worth it only when the number of steps and the path cannot be known in advance.
    The agent loop
    An agent is a loop: the model calls a tool, gets the result and decides the next step until it stops. Always set limits on steps, token budget and time, define clear stop conditions, and return tool errors to the model as observations so it can adjust.
    Designing tools for agents
    Few tools with clear names and descriptions, strict parameters, concise and informative output, pagination, and idempotent actions where possible. Tool definitions consume context, so too many tools make the agent choose wrongly and cost more.
    Model Context Protocol (MCP)
    MCP standardizes how apps (hosts and clients) connect to servers that expose tools, resources and prompts, over stdio for local use or Streamable HTTP for remote servers, with OAuth-based authorization. MCP now sits under the Linux Foundation's Agentic AI Foundation, so it is worth learning as a standard rather than a single vendor's feature.
    Human in the loop
    Irreversible actions such as sending email, deleting data or making payments need human approval. Combine that with least privilege per tool, a dry-run mode and an audit log of who approved what.
    Multi-agent trade-offs
    Subagents isolate context and let research branches run in parallel, but they multiply token usage and make debugging much harder. Start with one agent and split only when you can measure the benefit; for long-lived instructions, consider formats like AGENTS.md or Agent Skills that load only when needed.

    Do

    • Set limits on steps, tokens and time for every agent loop.
    • Require human approval before any write or irreversible action.
    • Trace every tool call with its arguments, result and duration.
    • Write evals for the agent's whole trajectory, not only the final answer.

    Watch out for

    • Giving an agent too many tools with vague descriptions.
    • Using multiple agents when a fixed workflow would do.
    • Granting an MCP server or tool broader permissions than the task needs.

    Checkpoint project: MCP server for your app's data

    Write an MCP server with three tools over your app's data (search posts, get a post, draft a summary) and an agent inside the app that uses it. It passes when the server works with MCP Inspector, the agent has a step limit, every tool call is traced, and write actions run only after the user confirms.

    • agents
    • Model Context ProtocolMCP
    • tool use

    Quick check

    0/4 answered · 0 correct

    Get at least 3 of 4 right to mark this stage as passed.

    1. Question 1 of 4IntermediateA task always runs the same steps: classify the ticket, fetch the account, draft a reply. Which design fits best?
    2. Question 2 of 4IntermediateWhy must every agent loop have limits on steps, tokens and time?
    3. Question 3 of 4IntermediateA tool call fails because the order id the model passed does not exist. What should your agent code do?
    4. Question 4 of 4IntermediateAn MCP server must be shared by many remote clients across the company. Which transport fits?

Advanced08–10

  1. 08Stage 08, 4 weeksProduction: observability, security, reliabilityRun AI features in production with tracing, guardrails, prompt injection defenses, rate limits and fallbacks.
    To me this is the part closest to backend work, with one new layer of risk: the data going into the model can itself be an attack. I learned how to think about prompt injection from security people, not on my own.

    What to learn

    Tracing and observability
    Follow the OpenTelemetry GenAI semantic conventions to trace every model call, retrieval and tool call with tokens, cost and latency. Tools like Langfuse let you replay the prompt, context and output and connect them to user feedback.
    Direct and indirect prompt injection
    Malicious instructions can come from the user or hide inside web pages, retrieved documents, emails and tool results. There is no complete fix yet, so the main mitigation is design: avoid combining private data, untrusted content and the ability to send data out in the same system (the lethal trifecta).
    OWASP risk lists
    The OWASP Top 10 for LLM Applications and the Top 10 for Agentic Applications are good checklists for design review: prompt injection, sensitive information disclosure, excessive agency, unbounded consumption. Use them as input for threat modeling and for a red-team eval set.
    Guardrails
    Prefer deterministic checks first: schema validation, allowlisted tools and domains, length limits, PII redaction. Only then add moderation or model-based classifiers for what is hard to express as rules.
    Rate limits, budgets and fallbacks
    Limit per user and per tenant on both requests and tokens, and set monthly cost budgets. Retry with backoff, fall back to another model or provider, use a circuit breaker, and keep a non-AI path for when everything fails.
    Data privacy
    Know exactly what data is sent to the provider, its retention policy and options like zero data retention or data residency. Do not write raw PII into traces and logs, and follow the personal data protection laws where your users are.

    Do

    • Trace every model call, retrieval and tool call using one consistent convention.
    • Apply least privilege to agents and keep untrusted content separate from instructions.
    • Set per-user rate limits and token budgets before opening a feature to the public.
    • Run a prompt injection red-team eval in CI.
    • Prepare a fallback provider and a non-AI path.

    Watch out for

    • Believing a line in the system prompt like ignore strange instructions is enough to stop injection.
    • Logging full prompts that contain personal data into the observability system.
    • Not capping tokens per user, so a single user can burn the whole budget.

    Checkpoint project: Harden the RAG and agent app

    Add tracing, per-user rate limits and token budgets, a fallback provider, and a 20-case injection red-team set (including malicious instructions hidden in retrieved documents) to the app from earlier stages. It passes when a dashboard shows cost per user, the injection pass rate is tracked in CI, and disabling the primary provider in a test environment makes the app fall back automatically.

    • observability
    • security
    • OpenTelemetry

    Quick check

    0/4 answered · 0 correct

    Get at least 3 of 4 right to mark this stage as passed.

    1. Question 1 of 4AdvancedAn email assistant reads the inbox, opens links in emails, and can send email. Why is this design dangerous?
    2. Question 2 of 4AdvancedA team's only defense against prompt injection is the system prompt line 'ignore any strange instructions'. What should they do?
    3. Question 3 of 4AdvancedOne user scripts thousands of requests to your AI feature and burns the month's token budget in a day. What prevents this?
    4. Question 4 of 4AdvancedWhy should you avoid logging full raw prompts into the observability backend?
  2. 09Stage 09, 3 weeksFine-tuning and small, local modelsJudge when fine-tuning or a small open-weight model is worth it, and fine-tune and serve one for a narrow task yourself.
    I have not fine-tuned anything for production; this stage is what I tried in side projects and learned from docs. The clearest takeaway is that fine-tuning teaches behavior and format, not new knowledge.

    What to learn

    When to fine-tune
    Consider it only when prompting, RAG and evals have plateaued, and when you need stable format or tone, or lower cost and latency for a narrow, high-volume task. Knowledge that changes often should still go through retrieval.
    LoRA, QLoRA and PEFT
    Instead of retraining all weights, LoRA trains small adapters so it is cheap and fast, and QLoRA adds quantization to run on GPUs with less memory. Data quality matters more than quantity, and always keep a held-out eval set to compare against the baseline.
    Distillation
    Use a large model's outputs to create training data for a small model on a specific task. Read the provider's terms of use and the model license carefully before doing it.
    Open-weight models and licenses
    The Hugging Face Hub is where you find and compare open-weight models; each model family has its own license, and some restrict commercial use or user scale. Read the license before putting a model into a product.
    Local inference and serving
    Ollama is convenient for local development, llama.cpp with the GGUF format lets you quantize to 4-bit or 8-bit trading quality for memory, and vLLM fits when you need high throughput. Most expose OpenAI-compatible endpoints, so switching between them is easy.
    Small models on devices and in the browser
    Small models do very well at classification, routing, embeddings and simple extraction, and can run in the browser through WebGPU with Transformers.js. Architectures that use a small model for simple steps and a large one for hard steps are becoming common.

    Do

    • Establish a prompting baseline on a hosted model before fine-tuning.
    • Compare on the same eval set for accuracy, latency and cost per thousand requests.
    • Read model licenses and provider terms before using their data or weights.

    Watch out for

    • Fine-tuning to add knowledge instead of using retrieval.
    • Looking only at benchmark scores and ignoring GPU operating cost and self-hosting complexity.

    Checkpoint project: Small ticket classifier

    Fine-tune a small open-weight model with LoRA to classify support tickets into 8 categories, then compare it with a prompted hosted model on the same eval set. It passes when you have a table of accuracy, latency and cost per thousand requests, and the model is served through Ollama or vLLM with an OpenAI-compatible endpoint.

    • fine-tuning
    • LoRA
    • open-weight

    Quick check

    0/4 answered · 0 correct

    Get at least 3 of 4 right to mark this stage as passed.

    1. Question 1 of 4AdvancedA team wants to fine-tune a model so it knows this week's price list. What is the better approach?
    2. Question 2 of 4AdvancedWhat does LoRA change compared with full fine-tuning?
    3. Question 3 of 4AdvancedBefore fine-tuning a small model for ticket classification, what should you establish?
    4. Question 4 of 4AdvancedWhy read an open-weight model's license before shipping it in a product?
  3. 10Stage 10, 2 weeksResponsible AI and governanceAssess the risks of an AI feature, document them in a system card and build the needed controls into the development process.
    At first I thought this was the legal team's job. In reality many of these decisions, like what data goes to a provider or whether a human must approve, live right in the code I write.

    What to learn

    Risk frameworks
    The NIST AI RMF splits risk management into govern, map, measure and manage, and is practical enough to apply per feature. Use it to answer who is accountable, what the possible harms are, how you measure them and how you respond.
    Regulation
    The EU AI Act classifies systems by risk level and adds transparency duties such as telling users they are interacting with AI and labeling AI-generated content, phased in over time. Check the rules where your users are too, for example Vietnam's personal data protection law.
    Fairness and quality per group
    Slice eval results by language, user group and input type to see where the model does worse, for example Vietnamese compared to English. A good average can hide a group that is served badly.
    Transparency and documentation
    Write a system card for the feature: purpose, out-of-scope uses, where data goes, eval results and known failure modes. Show clearly in the UI when content is AI-generated and consider C2PA for the provenance of generated media.
    Oversight and incident response
    Every AI feature needs an owner, a process for handling harmful output, and a kill switch behind a feature flag. Keep enough tracing to investigate an incident without violating privacy.

    Do

    • Write a short system card for every AI feature before release.
    • Add an AI risk checklist to the pull request template.
    • Report eval results per language and per user group.
    • Put every AI feature behind a feature flag that can be turned off immediately.

    Watch out for

    • Treating governance as a document written once and forgotten.
    • Reporting only the average score without looking at each user group.

    Checkpoint project: System card and checklist for the app

    Write a one-page system card for the RAG and agent app you built, plus an AI checklist in the PR template covering purpose, scope, which provider receives which data, eval results per language, known failures and the incident owner. It passes when the kill switch has been tested for real and eval results are reported separately for Vietnamese and English.

    • governance
    • responsible AI

    Quick check

    0/4 answered · 0 correct

    Get at least 3 of 4 right to mark this stage as passed.

    1. Question 1 of 4AdvancedOverall eval accuracy is 92%. Why also report results per language and per user group?
    2. Question 2 of 4AdvancedA newly launched AI feature starts producing harmful output. What should already be in place?
    3. Question 3 of 4AdvancedHow does the NIST AI RMF structure AI risk management?
    4. Question 4 of 4AdvancedYou are writing a system card for a new AI feature before release. What belongs in it?

Where the role is going

What matters right now, and where things are heading in the next few years.

Now

  • Evals are the core skill

    Serious AI teams have moved from trying things by feel to evals with datasets, graders and CI thresholds. LLM judges calibrated to each product's own criteria are replacing public benchmarks as the way to decide.

  • Context engineering matters more than prompt wording

    Quality depends heavily on putting the right information in at the right time: documents, memory, tool definitions, tool results. Knowing how to compact, summarize and load documents just in time is a highly valued skill.

  • Production agents need clear limits

    Agents are in production, but with step limits, least privilege, human approval and full tracing. MCP has become the common connection layer between models and tools or data.

  • Cost and latency engineering

    Prompt caching, batch APIs, routing between large and small models and per-user token caps are now routine work for an AI engineer. Cost per successful task is watched as closely as latency.

Next

  • Open agent standards are converging

    MCP, AGENTS.md, Agent Skills and A2A are being gathered under the Linux Foundation's Agentic AI Foundation. Learn the protocols and underlying concepts rather than tying yourself to one framework, because frameworks will keep changing fast.

  • Agent security becomes its own discipline

    As agents hold real permissions, problems like agent identity and authorization, goal hijacking and memory poisoning will need their own processes. Getting comfortable with threat modeling and red-team evals early is an advantage.

  • Stricter regulation and audits

    EU AI Act obligations keep phasing in and more countries have their own AI and data laws. Habits like writing system cards, keeping traces and evaluating per user group will become requirements rather than nice-to-haves.

  • Multiple models and small models close to users

    Systems will use several models at once, including small models running locally or on device for simple steps. Build a provider-agnostic architecture with evals so you can swap models in hours instead of weeks.

By the end of this map you can

  • You can estimate the cost and latency of an AI feature from real token counts before building it.
  • You can build LLM-backed APIs with streaming, structured outputs, tool calling, prompt caching and provider fallback.
  • You can build a RAG pipeline with hybrid search, reranking and citations, and measure retrieval recall.
  • You can design agents and MCP servers with step limits, least privilege and human approval steps.
  • You can build an eval suite with code-based graders and a calibrated LLM judge that runs in CI.
  • You can run AI features with tracing, rate limits, prompt injection defenses and a system card.

Related articles

All articles

Model Context Protocol in Practice: Building a Production MCP Server in TypeScript

MCP isn't magic, it's a well-designed JSON-RPC protocol. This post explains hosts, clients, servers, primitives and transports as of the 2026-07-28 spec, then builds a read-only TypeScript server over order data: tool design, pagination, testing with the Inspector, OAuth for remote servers, and the security holes you need to close.

Tooling & workflow19 min

Evaluating LLM Features: Knowing They Work and Keeping Them Working

Building a golden dataset from real usage, deterministic checks first, LLM-as-judge with rubrics and its biases, pairwise comparison, running evals in CI on prompt or model changes, tracking cost and latency alongside quality, online signals, tracing, and how to avoid overfitting your eval set.

Systems & architecture8 min

AI Engineer interview questions

See all 31

Other roles

Roadmap

Frontend Engineer

My main role

Building web interfaces that stay fast, accessible and maintainable as products grow

10 stages, 51 topics, ~34 weeks

Backend Engineer

Designs, builds and runs the APIs, data and services behind every product

10 stages, 52 topics, ~37 weeks

Mobile Engineer

Build iOS and Android apps that feel smooth, work offline and ship safely

10 stages, 50 topics, ~34 weeks

QA / Test Engineer

Find risk before users do and turn quality into something the team can measure

10 stages, 50 topics, ~36 weeks

DevOps / Platform Engineer

Builds infrastructure, pipelines and platforms so teams ship fast, safely and reliably

10 stages, 56 topics, ~42 weeks