SKILL.md
13,992 tokens · o200k_base · 57,747 bytes
Source excerpt starting at line 1.---name: n8n:create-instance-ai-evaldescription: >- Authors a new Instance AI workflow or Agent eval case — written locally as JSON, calibrated against a real build, then pushed to the LangTracer suite CI runs — build cases, behaviour/process cases, credential cases, and seeded (mid-conversation) cases — with intent-driven expectations. Use when adding or changing an Instance AI eval, or debugging why one is flaky.--- # Create an Instance AI eval Each eval is **one JSON case**. Author workflow cases in`packages/@n8n/instance-ai/evaluations/data/workflows/`. Author standalone Agentcases in `packages/@n8n/instance-ai/evaluations/data/agents/` and follow the[`create-agent-builder-eval` skill](../create-agent-builder-eval/SKILL.md).The disk loader auto-discovers `*.json` in both directories. A LangTracer suiteis the durable home. Cases validate against[`harness/schema.ts`](../../../packages/@n8n/instance-ai/evaluations/harness/schema.ts)(`.strict()` — unknown keys fail at load). The eval[README](../../../packages/@n8n/instance-ai/evaluations/README.md) is theexhaustive field reference; this skill is the opinionated *how*. > **Committing new case JSONs into the repo is no longer the recommended> approach.** Author the file locally (uncommitted), calibrate it against a real> build, then **push it to a lang-tracer suite** with `eval:langtracer-push`> (see [Push to a lang-tracer suite](#push-to-a-lang-tracer-suite)) —> `--suite baseline` for the consolidated corpus n8n CI runs, or a dedicated> capability suite like `agents`.> The suite is the home for the case; the eval CLI reads it back via> `--source langtracer`. You still write the JSON file — it's just the input to> the push, not a committed artifact.>> **Seeded cases.** An `inline` seed pushes with the case — the case-write API> stores it verbatim, so the suite is its home like any other case. Only a> `seed.mode: "replay"` case is refused (listed under `skipped:`): it's> reconstructed from a LangSmith trace at run time, so it dies when that trace is> pruned and has no durable home. Don't commit a replay case either — derive a> synthetic case from it. See [`case-shapes.md`](case-shapes.md). ## Set the autonomy level first **Before you source, draft, or run anything, decide how hands-on the driverwants to be — and say it back.** This skill runs at one of two autonomy levels.If the request makes the level clear ("just author and calibrate it yourself" vs."stop me at each step", or an explicit mode), adopt it, state it in one line, andnote how to override (e.g. "say 'stop me at calibration' to add a checkpoint").**If it's not clear, ask the driver one question** offering the two levels*before* doing any work. The skill has four natural decision **gates** — **selection** (which realfailure to encode), **shape + expectations** (archetype, must-haves, scopetrim), **calibration** (classify each red and resolve keep/loosen/drop), and**push** (kind + tier). The level decides what happens at each gate: | Level | Who decides when to stop | Behaviour ||---|---|---|| **autonomous** | agent | Runs all four gates start-to-finish; reports a **decision log** at the end for the driver to review — with the pushed case, its suite, and the source thread as **links** ([Share links, never bare ids](#share-links-never-bare-ids)); includes the observation id when the driver opted to record one; and includes a Linear ticket **proposal** for any kept capability-gap red ([Capability gap → propose a Linear ticket](#capability-gap--propose-a-linear-ticket)). || **checkpoint** | driver, per gate | Stops at each gate with a compact **proposal + recommendation**; driver says "go" or redirects. At the **calibration** gate, hands the driver a link to the just-built thread on the live instance plus login credentials so they can review the real conversation and workflow themselves before confirming (below). | **Calibration is special-cased at both levels.** A calibration verdict thatflips a case's *meaning* — a real capability-gap red vs. a harness-caused red, orany loosening that would let a known-bad build pass — is **surfaced explicitly**(interactively in checkpoint; in the decision log in autonomous), never silentlycommitted. It's the one call where a quiet mistake corrupts the suite, so itnever fully auto-commits. **Recording a source observation is optional, not a fifth gate.** After a realthread passes selection, offer to save why it was selected and what the developerobserved with LangTracer's `create_observation` tool (see[`sourcing-cases.md`](sourcing-cases.md#optional-record-the-selection-as-an-observation)).If the driver declines, has not stated a preference in autonomous mode, or thewrite fails, continue with the eval. Never block drafting, calibration, or pushon an observation. `create_observation` and `update_observation` write only toLangTracer; do not add a LangSmith feedback or sync step. **Checkpoint calibration — review the real thread on the instance.** Because thecalibration verdict is trust-critical, in checkpoint mode you don't ask thedriver to trust your reading of the run. You built the case against a liveinstance with `--keep-workflows` (step 4), so the thread and the workflow arestill there — hand the driver a direct link and let them look: - **Thread:** `<base-url>/assistant/<threadId>` — the exact conversation the case ran (the run prints the `threadId`; the built workflow prints as `BUILT (<id>)` and opens at `<base-url>/workflow/<id>`).- **Login:** the email + password the eval signs in with (the owner you seeded on the instance — see [`running-evals.md`](running-evals.md); the default local seed is `nathan@n8n.io` / `PlaywrightTest123`). Present, per red: the assertion, whether it went green/red, your proposedclassification (real capability gap / harness limitation / noise) andkeep/loosen/drop, and the review link. The driver logs in, reads the thread andthe workflow, and confirms or redirects before you write the verdict back intothe case `description`. ## Share links, never bare ids Every lang-tracer entity has a shareable web page, but the CLI and the MCP handyou **numeric ids** — `eval:langtracer-push` prints `+ created <slug> (#621)`,`get_eval_run` returns a run number, `list_conversations` returns thread ids.An id is unclickable: the driver has to go find it. **Whenever you name a case,suite, thread, cluster, or run in anything a human reads** — a checkpointproposal, the end-of-run decision log, a PR description, a Linear ticket, a Slackmessage — render it as a link, keeping the id in the label: ```pushed as [#621](https://lang-tracer.n8n-maintenance.workers.dev/test-cases/621)``` Build links off the **web base** (`LANGTRACER_URL`, in production`https://lang-tracer.n8n-maintenance.workers.dev`). Never off the API bases —`${LANGTRACER_URL}/api/v1` and `/api/mcp` are machine endpoints, and a link intoeither 404s for the driver or dumps JSON. | Entity | URL | Where the id comes from ||---|---|---|| Test case | `<base>/test-cases/<id>` | push output `(#<id>)`; `create_test_case` / `search_test_cases` || Suite | `<base>/suites/<suiteId>` | push header `Suite "<slug>" (#<id>)`; `list_suites` || Source conversation | `<base>/conversations/<threadId>` | `list_conversations` / `get_conversation` || Cluster report | `<base>/clusters/<id>` | `list_cluster_runs` / `get_latest_cluster_run` || Eval run (sweep) | `<base>/results?sweep=<runId>` | `list_eval_runs` / `get_eval_run` (`runId` *is* `sweeps.id`, the "run #N") | Two links that are **not** lang-tracer and don't take this base: the built thread(`<base-url>/assistant/<threadId>`) and workflow (`<base-url>/workflow/<id>`) liveon the **n8n instance** the eval ran against. When both are relevant — reviewing acalibration red, writing a capability-gap ticket — give both, labelled, so nobodyhas to guess which host a link points at. ## Where the best cases come from The strongest cases encode a **real** failure, not an invented premise. Twoconnections help you find and verify one: **LangTracer** clusters realconversations into capability-gap themes (discover what actually fails, atscale), and **LangSmith** holds the raw traces (verify exactly what happened in arun). LangTracer is the discovery layer; the durable artifact is almost always asynthetic case you author from what you learn (use `seed.mode: "replay"` only per[`case-shapes.md`](case-shapes.md)). See[`sourcing-cases.md`](sourcing-cases.md) for connecting the MCPs and thediscover → verify → encode workflow. ## Pick the case shape first The corpus is four archetypes. Decide which you're writing before you draft — itdetermines the fields, the grading, and how you validate. They compose (a seededcase can still assert outcome), but the primary shape drives the work. | Archetype | Question it answers | Primary fields ||---|---|---|| **Build** (default) | Does the workflow the agent builds actually *work*? | `outcomeExpectations` + `executionScenarios` || **Behaviour / process** | Does the agent *converse* correctly (ask the right clarifying question, not re-ask, honour a correction, respect plan approval)? | `processExpectations` + multi-turn director script; often **build-only** || **Credential** | Does the build behave correctly given a specific credential view? | `credentials[]` || **Seeded** | Start mid-thread, with prior work already in place, and drive the turn under test | `seed` (authored `mode: "inline"`; `"replay"` for a local check) | **Build** is documented in full below. The other three, the director-scriptvocabulary, and the seeding modes are in [`case-shapes.md`](case-shapes.md). ## Core principle (all shapes) **Write expectations from intent, then calibrate against a real build.** Decideup front what makes *any* correct solution correct — the must-haves implied bywhat the user actually said — then build the workflow once for real to calibrategranularity: loosen what's over-specified, confirm the must-haves areachievable, and catch requirements the agent legitimately satisfies a differentway. Don't transcribe one observed build into assertions — that overfits theeval into "did the agent reproduce that run" instead of "did it solve theproblem." **Keep the conversation in the user's voice.** State the goal and realconstraints the way a user would — don't name node types, wire up the structure,or restate your `outcomeExpectations` in the prompt. If the conversation spellsout the build, the case only tests whether the agent can follow instructions andthe expectations become tautological; the gap between what the user asks for andhow a correct workflow realizes it is the capability under test. Even when theanchor *is* honoring a user's stated technical preference, phrase it as theirneed + constraint ("I need field X and the built-in node doesn't expose it, sopull it straight from the API") — not as an implementation spec ("use an HTTPRequest node"). **Write the conversation in English** unless the user asked otherwise (or thecase exists specifically to test non-English handling). Sourced real threads arefrequently non-English — translate the intent into English when you rewrite theprompt in the user's voice; the failure mode is the anchor, not the originallanguage. **Trim to the smallest multi-turn conversation that reproduces the issue.**Real sourced threads are long (dozens of turns of setup, debugging, andtangents) — do **not** transcribe them. Distill to the fewest turns that stilldrive the build or behaviour under test. Every retained turn must earn its place:a turn stays only if it is *load-bearing* — a value the agent must ask for(withheld until asked, via a director note), a correction/push-back the caseexists to test, or a plan approval that gates the build. If removing a turndoesn't change what's tested, remove it. **Collapse to a single turn** wheneverthe whole request can be stated at once without a load-bearing exchange; keep itmulti-turn *only* for those exchanges, and keep each director script in one turn(don't fabricate assistant "done" turns to sequence steps — see[`case-shapes.md`](case-shapes.md)). A minimal conversation isolates thecapability; a transcribed one buries it in noise and tests instruction-following. **Size the build, not just the assertions.** Real sourced prompts are oftenkitchen-sink ("production-ready, runs forever, 3 feed posts *and* 8 stories aday", "generate 50 articles daily") and reliably blow the ~900s build budget (see"Known harness limitations"). A **faithful trim is a legitimate authoring move**:reduce batch sizes, drop one of several parallel pipelines, or merge adjacent AIsteps so the case builds within budget — then note the reduction in the case`description` ("the original request also asked for an 8-stories/day pipeline;scoped to feed posts so it builds in budget"). Keep the capability under test; cutthe combinatorial bulk. A case that never builds tests nothing. ## Workflow These steps map to the four gates from [Set the autonomy level first](#set-the-autonomy-level-first):sourcing (before step 1) is the **selection** gate, including the optional offerto record a source observation; steps 1–2 are the **shape +expectations** gate; steps 5–6 are the **calibration** gate; steps 7–8 are the**push** gate. In *autonomous* mode you flow through all of them and summarize ina decision log; in *checkpoint* mode you pause at each with a proposal, and atcalibration you hand the driver the thread link + login to review the real build(see [Set the autonomy level first](#set-the-autonomy-level-first)). Calibration(step 6) always surfaces meaning-flipping verdicts explicitly regardless of level. 1. **State the must-haves first.** From the conversation alone, list what every correct workflow must do (trigger type, essential operations, gating condition) — those become draft `outcomeExpectations`. Required fields: `conversation` (≥1 turn, first `user`), `complexity`, `tags`, and **at least one** of `executionScenarios` / `processExpectations` / `outcomeExpectations`.2. **Draft the case** from the template below; validate it loads (see "Validate").3. **Smoke-test the *environment* with one case before any batch.** Run a single case end-to-end first. This validates auth / model / `--base-url` / the built dist for ~1/Nth the cost — distinct from validating a *case*. If that one case crashes at execution (especially with an identical error you'd expect to hit every case), fix the environment before running the batch (see "A red is signal" → environment check). Running 15 cases only to discover a stale-dist crash on all of them wastes a full run.4. **Build it once** against a running instance (see [`running-evals.md`](running-evals.md)) with `--keep-workflows` so the built workflow stays for inspection.5. **Inspect** — read the built workflow (the run prints `BUILT (<id>)`; fetch via `GET /rest/workflows/<id>`) and the HTML report's transcript to see what the agent actually did.6. **Calibrate — sharpen assertions; never dull them to force a green.** Fix assertions that are genuinely mis-sized: relax one that pins a choice the conversation left open (so a valid *alternative* build wrongly fails), tighten one a wrong build would slip past, and phrase `executionScenarios` to match how the workflow runs on mocked data. But when a scenario goes red because the build has a real gap, or because the harness can't exercise it, **that red is the result — keep it and surface why** (see "A red is signal", below). Never delete a scenario, weaken an assertion, or drop to build-only just to make the run green. And when the case comes back **green**, that is a result to earn, not to accept: confirm the precondition actually fired, then re-derive it from the raw thread before calling the case a regression guard (see [First reproduce, then reclassify](#first-reproduce-then-reclassify)).7. **Push to the suite — do NOT commit the JSON.** Once calibrated, push the case into its curated lang-tracer suite with `eval:langtracer-push` (see [Push to a lang-tracer suite](#push-to-a-lang-tracer-suite)); the suite is the case's home, not the repo. Leave the `data/{workflows,agents}/*.json` file uncommitted (or delete it once it's in the suite). Committing new case JSONs into the repo is no longer the approach. (An `inline` seed pushes with the case; only a `replay` case is refused — it's a local throwaway; see [`case-shapes.md`](case-shapes.md).) For a sourced case, finish by **linking it to its source thread/finding** over the MCP — see [Link the pushed case to its source](#link-the-pushed-case-to-its-source-provenance-step--always-do-this).8. **Hand back links, and a ticket proposal if the case found a gap.** Report the pushed case as `<base>/test-cases/<id>`, not `#<id>` ([Share links, never bare ids](#share-links-never-bare-ids)), and if calibration kept a real capability-gap red, propose a Linear ticket for it ([Capability gap → propose a Linear ticket](#capability-gap--propose-a-linear-ticket)) rather than leaving the gap as a red case nobody owns. `--iterations N` is available to measure flakiness (pass@k / pass^k) — reach forit when you suspect a case is non-deterministic or before promoting it to agated tier, not as a routine step (each iteration is a full build + execution). Gut-check: if you can't picture a plausible *wrong* build that this casereliably turns **red**, the assertions are too loose to guard anything. **Confirm the precondition fired, not just the green.** For any *conditional*assertion — "when X happened, the agent did Y" (most `processExpectations`, andany behaviour case) — a pass has two readings: the agent did Y, or **X neverhappened** and the assertion passed vacuously. A behaviour case that hinges onthe mock producing a specific failure (e.g. an AI node simulated to empty so adownstream parse node fails) is the classic trap: if the mock instead returnsparseable data, the failure never occurs and the case guards nothing whileshowing green. Calibration must read the execution trace and the agent's`finalText` (`buildTrace.finalText` in the verifier snapshot, or the HTML report)and verify X actually materialised — the direct-loop `eval-results.json` does notpersist per-expectation judge reasoning, so pass/fail alone can't tell you whichreading you got. **The negative form is the easiest to fool yourself with.** An assertion phrasedas "the agent did NOT call `X` with a bad argument" passes when the agent called`X` correctly *and* when **it never called `X` at all**. Those are oppositeresults and the judge reports the same green. So for any assertion about toolmisuse, confirm the tool was actually invoked before believing the pass: parse`testCases[].transcriptPerRun[][].steps[]` for the call. Note the transcriptgroups multi-action tools under a bare `toolName` (`nodes`, `workflows`,`credentials`), so read `args.action` to get the real one — filtering on`nodes[explore-resources]` finds nothing and looks like a clean pass. Measured onthis corpus: a batch of five tool-misuse cases scored 100% on its firstcalibration run, and three of them were passing vacuously because the tool undertest was never called. **A sourced failure that does not reproduce is not yet a regression guard — firstre-derive the precondition.** Behaviour does drift across versions, and a casethat flips from *capability-gap* (red) to *regression guard* (green, catches are-introduction) is a legitimate and valuable outcome. But reach it byelimination, not by default: a green far more often means *your case never set upthe situation* than *the builder improved*. See below. ## First reproduce, then reclassify A case built from a real failure that comes back green is the most commonoutcome of a first calibration run, and "the build must have improved" is themost common wrong conclusion. The usual cause is that you authored from a*summary* of the thread — the theme label, the observation description, your ownone-line note — and the trigger you assumed is not the trigger that fired. Goback to the raw thread before you downgrade anything. **Find the turn, not the topic.** Locate the exact tool call that failed, thenread the assistant text immediately before it. The agent usually states itsintent in the open, and that sentence is the precondition. Then ask what *state*made that call necessary — not what the conversation was about. Worked example from this corpus. Sourced finding: "the agent invents`nodes[explore-resources]` method names." Assumed trigger: the user swaps modelprovider. A case built on a clean provider swap came back green — the agent setthe model id directly and never called the tool at all. The raw thread said itplainly: > "The **Groq Chat Model** has an invalid model (`llama3-8b-8192` isn't offered> by your Groq credential). **Let me list valid models and fix it.**" The precondition was never the swap. It was *an existing model id that theprovider rejects at runtime, with a credential already connected* — that is thestate that makes enumerating models necessary. Rebuilt on it, the same casereproduced the failure on the first run, with the agent inventing two methodnames in a row. Three moves turn a non-reproducing case into a reproducing one. Try them inorder before settling for a guard: 1. **Fix the precondition.** Rebuild the seed and the live turn to recreate the state the source thread was in, not the subject it was discussing.2. **Move the assertion to the first call.** A mechanism where the agent *self-corrects* grades green on the end state and is still a real defect — the wasted round-trip and the guessed schema are the finding. Grading first-call correctness turned a mechanism previously dismissed as "self-corrects, weak eval" into a gap that reproduced in 2 of 2 runs.3. **Keep what the attempt actually caught.** A reproduction run often reds on a *different* real defect than the one you targeted. That is still a capability-gap finding — keep the red, retarget the description, and say plainly in it that the originally targeted mechanism did not reproduce. **Know when to stop.** Some mechanisms are structurally unreachable in thisharness, and no amount of re-deriving fixes that. The clearest example: a seedrestores a **fresh** workspace file at the start of the graded turn, so theagent's first `old_str` always comes from a file it just read. Failures that need*accumulated drift* across many turns — the `str_replace` byte-fidelity family,the largest agent-caused tool-call failure in production at 15% of threads — donot reproduce even in a multi-turn chain of edits, each followed by a build. Capthe effort at about three attempts, then write the negative result into the case`description` as the finding it is, and ask whether the real defect belongs in aticket rather than an eval. ## A red is signal — surface it, don't work around it Calibration exists to right-size assertions, **not** to make a case pass. When arun turns a scenario or expectation red, classify the red first — then keep it. **First rule out the environment.** Before reading any red as a signal about acase, check the shape of the failures across the run. If **every case fails thesame way** — every scenario with the *same* execution error while builds succeed,*or* every **build** erroring identically before it starts (an `Agent error:Something went wrong…` with **zero tool calls**) — that is almost never the cases;it's a broken environment, most often a **stale dist** after a branch or worktreeswitch. Two shapes to know: - **Stale `packages/core` / `packages/cli` dist** — a refactor moved a runtime export and the built dist still calls the old one, so *builds succeed but every execution fails the same way* (e.g. `(0 , n8n_workflow_1.createDeferredPromise) is not a function` after `createDeferredPromise` moved to `@n8n/utils`).- **Stale/half-built `@n8n/instance-ai` dist** — every run errors *before building* (`Agent error…`, zero tool calls) and the instance log shows `Cannot find module '@/utils/...'` from `dist/skills/*.js`: the build's `tsc-alias` step (which rewrites `@/` path aliases to relative requires) didn't complete, so the dist is internally inconsistent. - **Out-of-sync `node_modules`** — `pnpm build` itself dies early with `Cannot find module '@n8n/<pkg>'` even though that package is a declared `workspace:*` dependency *and* has a `dist/`. The workspace symlink is missing from the consumer's `node_modules` (typical after a branch or worktree switch). Confirm with `ls -d packages/<consumer>/node_modules/@n8n/<pkg>`; fix with a plain `pnpm install` — no need for the heavier `pnpm reset --full`. Fix it, don't calibrate around it: run a full ordered `pnpm build` (a targeted`--filter` build can fail on unrelated stale-dep type errors; for the instance-aishape, `cd packages/@n8n/instance-ai && pnpm build` runs `tsc && tsc-alias`), then**restart the instance** — the running node process holds the old dist in memory,so rebuilding on disk changes nothing until restart (and `kill` by env-var patternmisses it — kill the actual `lsof -t -iTCP:<port>` PID). Re-probe one case, confirmit builds and executes, then re-run the batch. Only once uniform environmentfailures are excluded do the three categories below apply: - **Real build / capability gap** — the agent's workflow is wrong or missing something the user asked for (a miswired branch, a missing retry, wrong field keys). This is exactly what the eval is for. **Keep it red.** Don't loosen the assertion or drop the scenario; a currently-red gap is the capability signal today, and a re-introduction guard once the builder improves. Then **propose a Linear ticket** for the gap — see [Capability gap → propose a Linear ticket](#capability-gap--propose-a-linear-ticket).- **Harness limitation** — the build is correct but the mock/execution layer can't exercise the path (see "Known harness limitations", below). **Keep the scenario and say so in its `description`** — that this red is harness-caused, not a build defect — so nobody misreads it as a product bug. Keep it out of gated tiers if it hard-fails every run; when the harness gains the capability it starts earning its keep with no re-authoring.- **Genuine non-determinism** — the *same* build flips green/red across runs. This is the only real "noise". Confirm it with `--iterations N` before calling it flaky, then de-tier and note it; deletion is the last resort. **Annotate every kept red in the case `description` with a scannable prefix** so afuture reader tells the two apart at a glance. Use `Harness note: …` for aharness-caused red (name the limitation and why the build is still correct), and`Capability-gap finding: current build reds because <X> — a real builder bug(flips to a regression guard once fixed)` for a real gap. Consistent prefixes keepthe corpus greppable and stop harness reds from being misread as product bugs. The one move to never make is **working around a red by weakening what the casechecks** — deleting a failing scenario, loosening an assertion until a wrongbuild would pass, or quietly converting to build-only. That makes the suite lookgreener than the product is, which is the opposite of the eval's job: bugs andharness gaps are the deliverable, so **highlight them, don't engineer aroundthem**. If you catch yourself editing a case so that a known-bad build would nowpass, stop. **Who confirms the classification depends on the autonomy level.** In*checkpoint* mode the keep/loosen/drop decision is the driver's to confirm: youhand them the thread link + login (see [Set the autonomy levelfirst](#set-the-autonomy-level-first)) so they can review the real conversationand workflow, then you write the agreed `Harness note:` / `Capability-gapfinding:` prefix back into the case `description`. In *autonomous* mode the agentproposes it explicitly in the end-of-run decision log. Either way theclassification is stated in the open, never silently committed — misreading aharness red as a real gap (or the reverse) is the one calibration mistake thatquietly corrupts the suite. ### Capability gap → propose a Linear ticket A kept capability-gap red is a **product bug you just characterised better thanany bug report would**. But a red case in a suite doesn't assign itself to anyone:without a ticket the gap sits in CI as permanent noise, and the next person toread the run assumes someone already owns it. So once a red is classified as areal gap (and the driver has confirmed it, per the autonomy level), **propose aLinear ticket for it.** **Propose, don't create.** Per [AGENTS.md](../../../AGENTS.md), never open aLinear ticket unasked. Put the draft in front of the driver — interactively incheckpoint mode, in the decision log in autonomous mode — with a title, a team,and the body, and let them say go. Skip the proposal in two cases: - **The gap already has a ticket.** Check the case's linked issues on its page, and run `get_linear_ticket_context <TEAM-N>` on any candidate identifier the driver or the source thread mentions, before you draft a duplicate.- **The red isn't a capability gap.** A `Harness note:` red is a lang-tracer/harness issue, and genuine non-determinism is a case-hygiene chore. Neither belongs in the builder's queue. The draft body should carry what makes the gap actionable, all of it already inhand from calibration: - **The eval case**, as a link — `<base>/test-cases/<id>` (see [Share links, never bare ids](#share-links-never-bare-ids)). This is the reproducer; it's the most valuable line in the ticket.- **What failed, verbatim** — the failing `outcomeExpectation` / `processExpectation` or scenario name, plus the judge's stated reason. Not a paraphrase: the exact text is what the fixer will grep for.- **What the build did instead** — the specific defect (miswired branch, wrong field key, missing gate), and links to the real evidence: the source conversation (`<base>/conversations/<threadId>`) and the built thread + workflow on the eval instance (`<base-url>/assistant/<threadId>`, `<base-url>/workflow/<id>`).- **Blast radius, if you know it** — the cluster theme or the number of real conversations behind the gap (`<base>/clusters/<id>`) is what turns "one red case" into a prioritisable bug. **File it from the case page so the link is made.** The `<base>/test-cases/<id>`page has a *Create Linear issue* dialog that creates the ticket **and** links it tothe case; that link is what makes `get_linear_ticket_context <TEAM-N>` later returnthe case, its scenarios, the source conversation, and its analysis in one call. Thecase↔ticket link is only writable from that UI — lang-tracer's MCP and `/api/v1`are read-only for it — so if the ticket gets created some other way (a Linear MCP,if your harness has one, or Linear directly), say plainly that it isn't linked, askthe driver to link it on the case page, and meanwhile put the identifier + URL inthe case `description` via `update_test_case` so the provenance isn't lost. Then extend the description prefix with the ticket, so the corpus stays greppablein both directions: `Capability-gap finding: current build reds because <X> — areal builder bug (flips to a regression guard once fixed). Tracked in[<TEAM-123>](<ticket url>)`. And when the *build itself* is wrong — not just ascenario red under a correct build — push it with `--set-kind capability_gap` intoa suite of that kind (see[Push to a lang-tracer suite](#push-to-a-lang-tracer-suite)). ## Example Minimal build case: ```json{ "description": "What this case tests.", "conversation": [{ "role": "user", "text": "<the build prompt>" }], "complexity": "medium", "tags": ["build", "<nodes>", "<concepts>"], "triggerType": "schedule", "outcomeExpectations": ["<a must-have any correct workflow satisfies>"], "executionScenarios": [ { "name": "happy-path", "description": "<what this run exercises>", "dataSetup": "<what the external services return>", "successCriteria": "<observable proof the run succeeded>" } ]}``` A fuller case with a multi-turn director script (withhold a value until asked,push back on a wrong plan): ```json{ "description": "Scheduled GitHub-bugs digest to Slack. Repo and channel are withheld until the agent asks; the plan must filter to the 'bug' label before it's approved.", "conversation": [ { "role": "user", "text": "Every weekday at 9am, fetch this week's open bugs from our GitHub repo and post a short summary to Slack." }, { "role": "assistant", "text": "Which repo and which Slack channel should I use?" }, { "role": "user", "text": [ "[Withhold the repo and channel until the agent asks; then say the repo is 'acme/widgets' and the channel is '#eng-bugs'.", "When the agent shows a plan or setup card, reject it unless it filters issues to the 'bug' label — a digest of ALL issues is wrong. Once it filters to bugs, approve.]" ] } ], "messageBudget": 8, "complexity": "medium", "tags": ["behaviour", "build", "schedule", "http-request", "slack"], "triggerType": "schedule", "processExpectations": [ "The agent asked for the repo and Slack channel before building, since the prompt named neither.", "The agent's final plan filtered issues to the 'bug' label — if its first attempt didn't, it corrected after the user pushed back rather than summarizing all issues." ], "outcomeExpectations": [ "A Schedule Trigger runs the workflow on a recurring weekday-morning cadence.", "Open issues are fetched from GitHub (HTTP Request or GitHub node) and filtered to the 'bug' label before the summary is built.", "One Slack message summarizing the fetched bugs is posted to the #eng-bugs channel the user gave." ], "executionScenarios": [ { "name": "posts-bug-digest", "description": "Three open bugs are returned; a summary is posted to Slack", "dataSetup": "The GitHub issues request returns three open issues labelled 'bug' ('Login 500', 'Timezone off by one', 'CSV export truncates'). The Slack postMessage call returns { \"ok\": true, \"ts\": \"1700000000.0003\" }.", "successCriteria": "The run completes without errors and posts one Slack message to #eng-bugs that references the three fetched bug titles." } ]}``` What each piece is doing: - **`conversation[0]` is sent to the builder raw.** The opening turn is the real prompt — never put a `[director note]` in it (it would leak verbatim).- **The `[bracketed]` turn is a director script** for the user-proxy — behaviour, never spoken. Here it withholds values until asked and rejects a plan that misses the label filter. Keep the whole script in one turn and encode ordering inside it (don't fabricate assistant "done" turns to sequence steps — see [`case-shapes.md`](case-shapes.md)). `applies-each-change-when-asked` (in the `baseline` LangTracer suite) is a good real example.- **`dataSetup` describes only what external services return.** That's the layer the harness controls (below). ## `dataSetup` and the mock layer The harness mocks by **intercepting outbound HTTP requests to external services**and having an LLM answer them from the node's config and API docs. It does **not**let you set a node's output directly, and it does **not** mock n8n internals(Code/Set/Merge/IF/Switch run for real on the mocked data; triggers and DBnodes get LLM-generated pin data). So: - Write `dataSetup` as **what each external service returns** ("the GitHub request returns three issues labelled bug"), not as node outputs or internal state.- The strongest scenarios exercise **external-service responses** — that's what the harness reproduces most faithfully.- **Data Table *reads* are pinned to the scenario.** A read op (`get` / `rowExists` / `rowNotExists`) is treated as the scenario's "stored state" and pinned with data derived from your `dataSetup`, so change-detection / dedup / "last seen" scenarios *can* be exercised — describe the stored rows in `dataSetup`. Two caveats: the pinned rows are LLM-generated (steered, not byte-exact — don't assert exact values off them), and *writes/inserts* aren't pinned (they hit the real per-thread table, recreated schema-only with **no rows**), so read-after-write within one run isn't faithful — the read reflects `dataSetup`, not what the run just wrote. A third caveat: only Data Table *reads* are seedable this way — **dedup / change-detection built on workflow static data** (`removeItemsSeenInPreviousExecutions`, `$getWorkflowStaticData`) is **not** seedable, because static data starts empty every run, so such a scenario reds vacuously (it sees everything as "new"). To get a seedable change-detection scenario, steer the build toward a Data Table; otherwise accept the static-data red as a harness limit and carry the logic in `outcomeExpectations`. Note the agent may *choose* static-data dedup on its own.- Don't assert exact counts that depend on mock generation ("exactly 7 posts"). Say "fewer than the original 10". ### Known harness limitations that turn scenarios red regardless of the build These produce a **reliable** red on a *correct* build. Don't engineer around them— write the scenario for the behaviour you want and note in `description` that thered is harness-caused (per "A red is signal", above): - **Resource-locator fields left empty for setup** (Google Sheets / Drive / Calendar and similar node pickers). The agent legitimately leaves the document/folder/calendar ID blank for the user to pick at setup; the mock substitutes `__evalMockResource`, and the node then crashes looking it up ("Sheet with ID __evalMockResource not found", or "Cannot read properties of undefined"). Any scenario whose success path runs *through* such a node hard-fails before anything downstream executes.- **Trigger and Data-Table-read pin data is *LLM-generated*, so not byte-exact.** Both are steered by your `dataSetup` (see the mock-layer section above — you *can* influence what a trigger emits or what a stored-row read returns), but because the values are generated, a scenario that asserts exact values or counts off them is flaky. Assert shape/branch/relative facts, not exact figures. (The residual hard red here: polling / form triggers still occasionally fail to load entirely — "workflow not found".)- **Mock response shape** — the LLM-generated mock response can omit the real envelope, crashing a downstream parse/format node. Recurring, reproducible shapes to expect (all produce a red on a *correct* build): - **OpenAI structured output** — historically the mock returned a plain `{content: "..."}` instead of the Responses envelope (`output[0].content[0].text`), so a **Structured Output Parser** / **Information Extractor** / **Text Classifier** got nothing and errored with **`Model output doesn't fit required format`**. The Responses-envelope normalizer (PR #33578, merged) fixes the flat-envelope case, so many of these now execute cleanly. A **narrower residual red remains** for structured-output schemas declared with strict **`additionalProperties: false`**: the normalized `output` wrapper (and any extra fields the mock invents, e.g. `subject`/`date`) violate the strict schema, so the node still rejects the mock output. Both the old and residual forms are the *mock*, not the build — carry correctness in `outcomeExpectations` and note the red as harness-caused. - **Gmail** mock returns headers as top-level capitalized fields (`From`, `Subject`) instead of under `payload.headers`, so a Code/Filter node reading the sender/subject gets empty strings (e.g. a "drop no-reply senders" safety gate lets everything through). Assert the *wiring/ordering* of such a gate in `outcomeExpectations`, not its runtime effect in a scenario. - A less-common API (e.g. Gemini's top-level `candidates`) can omit its envelope the same way. - **Google Drive resumable upload** — the initiate-upload mock omits the `Location` header carrying the session URL, so a Drive file-upload node fails with a 400. Any build that uploads a generated image/file to Drive can red on this.- **Agent-tool nodes can't be executed standalone.** An AI-Agent *tool* node (`toolHttpRequest` and other `supplyData`-only LangChain nodes with no `execute` method) only runs when the agent invokes it; the harness executing it directly fails with `has a "supplyData" method but no "execute" method`. A near-universal red for chat-trigger / AI-agent build cases whose tool is an HTTP-request tool — the build is correct, so carry correctness in `outcomeExpectations` (agent wired to trigger + model + tool) and note the execution red as harness-caused.- **Poll/wait loops can't be fast-forwarded.** A workflow that submits an async job then polls for completion (generate → poll status until ready → download) can't advance the mocked status deterministically, and a `Wait` node runs in real time, so the scenario reds with an execution timeout (`framework_issue`). The build is correct — carry it in `outcomeExpectations` and note the red as harness-caused.- **A build can time out and produce no scored result at all** — the run reports `BUILD FAILED: Run timed out` and zero graded expectations. Don't assume "spec too big": the more common cause is a **single-prompt case where the agent asks a clarifying `ask-user` question** and the build hangs on the unanswered question until the per-iteration timeout (see [`case-shapes.md`](case-shapes.md) — only confirmations auto-approve). Before treating a timeout as spec size, **classify it**: read the agent's final response in the report / trace (did it ask a question? flag an infeasibility? or genuinely churn through a huge build?), and **re-run the case solo (`--concurrency 1`)** — concurrency both masks a stalled build (it hits the cap) *and* can time out a perfectly healthy build purely by queueing it behind the per-instance build cap (default 4), so a solo run either surfaces the real reason in seconds or simply passes outright. Fix per cause: a solo run that now passes → it was **concurrency contention**, not the case (split big batches across lanes — see [`running-evals.md`](running-evals.md)); a clarifying-question stall → author multi-turn with a director note that pre-answers it; a genuine infeasibility → it's an infeasibility/honesty behaviour case (`processExpectations`), not a build case; a true oversized spec → the timeout is itself a finding, but note it so the zero isn't mistaken for a scored failure. ## outcomeExpectations vs processExpectations Both are natural-language assertions graded by the same Sonnet judge, and each**counts as a pass-rate unit**. They judge different surfaces: - **`outcomeExpectations`** — the **resulting workflow**, judged from the workflow JSON. Assert node choices and configuration, connection topology and branch wiring, data/expression references, trigger cadence, gating conditions. They run everywhere, including prebuilt/MCP runs (no transcript needed).- **`processExpectations`** — **how the agent behaved during the build**, judged from the transcript. Assert clarifying questions asked (or not re-asked), tool-call behaviour, plan/approval handling, batching, honouring a correction, ordering. They need a transcript, so they're **skipped in prebuilt/MCP runs**. Rule of thumb: an assertion about *the artifact* is an outcome expectation; anassertion about *the conversation or the agent's choices along the way* is aprocess expectation. A case with **no** `executionScenarios` is a valid**build-only** case, graded by these expectations plus the workflow checks. ## Sizing each assertion Right-size against **what the agent was actually told**. An assertion iswell-sized when every correct build passes it and a wrong or lazy build failsit — and it holds the agent only to what the conversation specified, not to onerun's arbitrary choices. Two failure modes: - **Too tight** — pins a choice the conversation *left open*. If the prompt never named a vendor, "calls flightaware.com" fails a valid build that used a different source. **But if the conversation specified it, pin it** — when the user said "email me via Gmail," "sends via a Gmail node" is correct and *required*, not too tight.- **Too loose** — a non-solution would also pass. "Fetches data from somewhere" passes a workflow that fetches but never compares — it doesn't prove the change-detection the prompt asked for. Quick check — the *substitution test*: would a reasonable alternativeimplementation *of what the user asked for* still pass? Examples (flight-statuscase, where the source and channel were **left unspecified**): | Verdict | Assertion | Why ||---|---|---|| ❌ too tight | "Has an HTTP Request node calling `flightaware.com`" | Vendor was unspecified; a valid AeroDataBox build fails. (If the user *had* said "scrape FlightAware", this would be correct.) || ❌ too tight | "Publishes via HTTP Request nodes" | Pins the *transport* when a first-party node is the idiomatic path — e.g. the Facebook Graph API node is the correct way to reach the Instagram Graph API, so a valid build using it fails. Assert the capability ("publishes to Instagram via the Graph API, through the Facebook Graph API node or HTTP Request"), not the mechanism. || ❌ too loose | "Fetches flight data from somewhere" | A workflow that fetches but never compares passes — doesn't prove change-detection. || ✅ right | "Persists the previously-seen status and compares it to the freshly-fetched one" | The defining behaviour; substitution-proof across vendors and storage choices. || ✅ right | "Alert is sent only on the change-detected branch, gated by a conditional" | Proves the gate without pinning node or channel. | Put intent the conversation only *implied* (a preferred but unstated channel) in`processExpectations`, not `outcomeExpectations`. ## Robust design vs harness flakiness Two different things — keep them apart: - **Robust assertion design (always do this).** The agent's unspecified choices vary run to run. Source-agnostic `outcomeExpectations` for an unspecified source aren't a concession to flakiness — they're the *correct* assertion.- **Harness limitations (surface them, don't hide them).** Some paths hard-fail on a correct build regardless of `dataSetup` — empty resource-locator fields that crash Sheets/Drive/Calendar nodes, polling triggers failing to load (see "Known harness limitations" above). (State-bearing Data Table *reads* are no longer in this bucket — they're pinned from `dataSetup`; only the write path and exact-value assertions stay unreliable.) The fix is to *document*, not to *work around*: note the limitation in `description` and keep a hard-failing scenario out of gated tiers. Only when a scenario flips **non-deterministically** run to run is it genuine noise worth removing — a scenario that reliably fails for a documented harness reason is a standing record of what the harness can't yet test, and stays. ## Negative execution scenarios Don't stop at the happy path — but only assert graceful handling the promptactually implied. Most agent-built workflows don't add error handling bydefault, so "the workflow crashes on bad input" is a legitimate builder finding,not a test-case bug. Where graceful handling *is* expected, phrase`successCriteria` as the *absence* of the wrong action ("no alert is sent", "runcompletes without error") as much as the presence of the right one: empty /not-found source, source error / timeout, malformed response. ## Relationship to the always-on workflow checks Every successful build is also graded by ~28 always-on binary checks across 7dimensions (structure, topology, parameter correctness, intent, AI wiring,craftsmanship, security) —[`binaryChecks/checks/`](../../../packages/@n8n/instance-ai/evaluations/binaryChecks/checks).Those are broad and low-visibility. **Writing a targeted expectation for yourspecific case is still worth it even when a binary check nominally covers it** —a named case-level assertion gives far better visibility into *this* behaviourthan one row buried in a 28-check rubric. Don't skip an assertion just because ageneric check exists. When a scenario fails, the verifier tags a **failure category** (`builder_issue`,`mock_issue`, `framework_issue`, `verification_failure`, `build_failure`). Treatit as a **hint, not ground truth** — we've seen a genuine node misconfigurationtagged `mock_issue`, and a real mock problem tagged as a build error. Open theHTML report and check the actual execution and the generated workflow beforeconcluding whether the failure is your case, the build, or the harness. ## Outputs of a run - **`workflow-eval-report.html`** (in the run's `.data/` dir) — the highest-value view: full conversation transcript with tool calls, per-node execution traces, the exact intercepted requests and the mock responses, Phase-1 hints, verifier reasoning, and the workflow-check rubric. Human-oriented; start here when debugging.- **`eval-results.json`** — structured results (the machine-readable artifact; the direct loop produces this even with no LangSmith). Good for an LLM or script to parse. **For per-case attribution under concurrency, parse this, not the streamed verbose log** — with more than one lane the log lines interleave across cases, so a `[scenario] FAIL` line in the stream can't be reliably tied to its case. Authoritative fields: `testCases[].buildSuccessCount`, `buildExpectationResultsPerRun[][].{pass,reason}`, and `scenarios[].runs[].{passed,failureCategory,rootCause,execErrors}`.- **`eval-pr-comment.md`** — the rendered PR comment (aggregate + regression comparison), always written. ## Validate (before running) ```bashcd packages/@n8n/instance-aipnpm exec tsx -e "import {loadWorkflowTestCasesWithFiles} from './evaluations/data/workflows/index.ts'; console.log(loadWorkflowTestCasesWithFiles('<slug>')[0].fileSlug)" # For a standalone Agent case:pnpm exec tsx -e "import {loadAgentEvalTestCasesWithFiles} from './evaluations/data/agents/index.ts'; console.log(loadAgentEvalTestCasesWithFiles('<slug>')[0].fileSlug)"``` ## Push to a lang-tracer suite Once a case is calibrated, push it (and any others) up into a curated lang-tracersuite instead of committing the JSON. `eval:langtracer-push` **upserts** over theREST API: it creates cases missing from the suite, updates ones whose contentdrifted, leaves the rest unchanged, and never prunes. It's the inverse of`--source langtracer` (which pulls a suite down). ```bashcd packages/@n8n/instance-ai# preview first — no writes:pnpm exec dotenvx run -f .env.eval -- pnpm eval:langtracer-push --suite baseline --dry-run --changed# then push (drop --dry-run):pnpm exec dotenvx run -f .env.eval -- pnpm eval:langtracer-push --suite baseline --changed``` - **Selectors** (at least one required — no accidental push-all): positional `<slugs...>` (exact file slugs), `--changed` (new/untracked + staged + modified `data/{workflows,agents}/*.json`, ideal right after authoring an uncommitted case), `--filter`/`--tier` (with `--exclude` as a modifier).- **Multiple positional slugs? Skip pnpm — call the script directly.** `pnpm eval:langtracer-push … slugA slugB` forwards the slugs as one joined argument (`"slugA slugB"`), so no case file matches and nothing is pushed. Either use a no-positional selector through pnpm (`--changed`), or run the script directly so each slug is its own argv: `pnpm exec dotenvx run -f .env.eval -- pnpm exec tsx evaluations/cli/langtracer-push.ts --suite <slug> <slug1> <slug2> …`.- **Env:** `LANGTRACER_URL` + `LANGTRACER_API_KEY` (an `lt_` bearer; one key works for MCP + REST) — put them in `.env.eval` and run under `pnpm exec dotenvx`.- **Options:** `--set-kind regression|capability_gap` (default `regression`, must match the suite's kind), `--contains-user-data` (default is `synthetic`). A case whose **build is correct** (outcome expectations green) but that carries a **currently-red execution scenario** from a builder bug is still a `regression` case — it guards the fix; reserve `capability_gap` for cases where the *build itself* is wrong.- **Scenarios sync on update too:** `PATCH /cases/:id` reconciles `executionScenarios` by name (update in place, insert new, delete missing — lang-tracer #48), so scenario edits re-push like any other field. A lang-tracer deployment predating that change silently ignores the key; if a pushed scenario edit doesn't land, update the scenario in the lang-tracer UI.- **Report what was pushed as links, not `#ids`.** The CLI prints `+ created <slug> (#621)` / `~ updated <slug> (#621, rev 3)` and a suite header — that's the id, and nothing more. Turn each one into `<base>/test-cases/<id>` (and the suite into `<base>/suites/<suiteId>`) in whatever you hand the driver, so they can open the case they just authored instead of hunting for it. See [Share links, never bare ids](#share-links-never-bare-ids).- **An `inline` seed pushes with the case:** the case-write API stores it verbatim, so a seeded case lives in a suite like any other. Only a `replay` case is refused — the push lists it under `skipped:`, because it's reconstructed from a LangSmith trace at run time and dies when that trace is pruned or deleted. Don't commit a replay case either; derive a durable synthetic case as the artifact instead. ### Link the pushed case to its source (provenance step — always do this) A sourced case that isn't linked back to the conversation/finding it encodes isan orphan: six months later nobody can tell what real failure it guards. Thepush CLI doesn't carry provenance, so after pushing, link the case over thelang-tracer MCP with one **`update_test_case`** call on the new case id (thepush prints it): 1. **`sourceThreadId`** — the source conversation's thread id (plus **`sourceRunId`** when the case anchors to one specific run/step within it). This is the DB-level link every by-version rollup, conversation float, and `?sourceThreadId=` query joins on — the tags/description below are the human-readable layer on top, not a substitute. The thread must already be imported into lang-tracer (running `get_conversation_analysis` on it, as the sourcing flow does, is enough); `source_kind` is derived server-side, and the link is only editable on authored cases — promotion-recorded provenance is immutable.2. **`expectedBehavior`** — the rule the case enforces, one paragraph — and **`failurePattern`** — what actually happened in the source thread, with turn references. Copy/adapt these from the analysis's `extractedCases` entry when the case came from `get_conversation_analysis`. Then **`add_case_tags`** (additive; targets the LT-side `tags` array, not`evalTags`, so nothing round-trips into eval runs): add a capability tag (e.g.`instruction-persistence`). Tag normalization is aggressive (lowercase, kebab);colon-form tags get silently dropped. And keep the thread id + turn refs inthe case `description` too (the drafter's habit of "Sourced from thread <id>"is the convention) — the description is the only field shown everywhere. ## Running You need a running n8n instance with Instance AI enabled and a working sandbox;point the eval at it. The harness runs in three modes — **direct loop** (noLangSmith; `eval-results.json` only), **LangSmith** (also records an experiment+ regression comparison), and **prebuilt** (`--prebuilt-workflows`, scoreexisting workflows). Narrow a run with `--filter <slug>` / `--tier <name>` /`--exclude`. See [`running-evals.md`](running-evals.md) for the run recipes,parallel lanes, tiers, and baselines, and the[README](../../../packages/@n8n/instance-ai/evaluations/README.md) for the fullflag list. Run with `--keep-workflows` when you want to review a build by hand —in *checkpoint* mode calibration this is how the driver opens the built thread(`<base-url>/assistant/<threadId>`) and workflow on the instance. ## Other eval harnesses (not this skill) Use the [`create-agent-builder-eval` skill](../create-agent-builder-eval/SKILL.md)for standalone Agent cases. Three other harnesses have their own data dirs andCLIs: **`eval:subagent`** (workflow-build compatibility corpus,binary-check scored), **`eval:discovery`** (asserts first-hop tool/dispatchrouting, no n8n server), **`eval:pairwise`** (head-to-head build comparison vs`ai-workflow-builder.ee`). Authoring them is out of scope here — see the READMEsections of the same names.
Discovery context
Discovered by repository scan. No exact path reference found in the snapshot’s root AGENTS.md.