n8n:create-instance-ai-eval

Source excerpt starting at line 1.

1# Running evals2 3A run does four things: **build** (Instance AI builds the workflow on a live4instance) → **Phase 1** (generate mock data hints) → **Phase 2** (execute with5external HTTP requests LLM-mocked) → **verify** (LLM grades `successCriteria`).6This file describes what the harness offers and how to point it at an instance;7the [README](../../../packages/@n8n/instance-ai/evaluations/README.md) has the8exhaustive flag list.9 10## What a run needs11 12- **A running n8n instance with Instance AI enabled**, reachable over HTTP. The13  eval is a client — it logs in and drives the normal build flow. Point it with14  `--base-url` (defaults to `http://localhost:5678`); use whatever instance you15  already run for Instance AI dev.16- **A login** — the eval signs in as an existing user. Set the eval login env17  vars to a user your instance has; the model and most settings have defaults18  that a working Instance AI dev setup already provides.19- **A working sandbox** — the build executes its workflow-build code in a20  sandbox, so the instance must have one configured (see below).21- **A model key for the eval helper** — mock generation, verification,22  user-proxy, and the expectations judge call an Anthropic-capable model.23 24## The sandbox25 26The build runs generated code in a sandbox that must be enabled and reachable on27the instance. How it's provisioned is a property of your instance, not the eval:28 29- **Via the hosted proxy** — the instance vends sandbox access (and can vend the30  model) through the AI-assistant proxy.31- **Direct** — the instance talks to the sandbox provider (Daytona) and the model32  provider with your own keys, bypassing the proxy.33 34Either path has a ceiling a parallel run can hit — the **proxy** enforces a35per-tenant quota, and going **direct** shifts that ceiling to your model36provider's rate limits. Neither is "the" setting; pick per what you're running37and lower `--concurrency` if you hit a limit. Configuration specifics live in the38instance's Instance AI config and the README's environment-variables section.39 40## Run modes41 42| Mode | How | Produces |43|---|---|---|44| **Direct driver** | no `LANGSMITH_API_KEY` | `eval-results.json` + HTML report locally — same pipeline and row order as the LangSmith driver (TRUST-261), row concurrency follows `--concurrency` |45| **LangSmith** | `LANGSMITH_API_KEY` set | also records an experiment and auto-compares against the baseline |46| **Prebuilt** | `--prebuilt-workflows <manifest>` | skips the build; verifies existing workflows (score MCP/hand-built cohorts on the same verifier) |47 48Narrow any run with `--filter <slug>` (filename substring, comma = OR),49`--tier <name>`, and `--exclude`. `--keep-workflows` leaves built workflows for50inspection; `--iterations N` runs each case N times for pass@k / pass^k.51 52**Seeded cases and `--keep-workflows`.** A seeded case's live turn addresses its53workflow the way a user would — by name, often loosely ("the batch image54workflow"). So a leftover copy is something the agent can rationally pick instead55of its own, and it prefers the one with failed executions when the message56mentions a failure; the judge then grades a different workflow than the agent57edited. That produces false greens as readily as false reds, so it doesn't58announce itself.59 60Restore now defends against this on both sides: each restored workflow gets a61`[seed <8 hex>]` name suffix so copies are distinguishable, and any leftover62carrying that suffix with the same base name is deleted before the next restore.63You'll see `Evicted N leftover seed workflow(s) before restore` when it fires.64Workflows without the suffix — real ones, and anything the agent built — are never65touched. So `--keep-workflows` is safe to use on a seeded case; the leftover is66cleaned up by the next run rather than contaminating it.67 68**Seeded agents are not evicted** — they get a fresh id per run but keep their69authored name, so `--keep-workflows` on an agent-seeding case leaves one behind70and they accumulate under the same name. That can't misdirect a later run (the71live turn is bound to its own agent by id), but it does clutter what the `agents`72tool lists. Delete them yourself when calibrating:73`DELETE /rest/projects/<projectId>/agents/v2/<agentId>`.74 75## Case source: disk vs langtracer76 77| Source | When to use it |78|---|---|79| **`disk`** (default) | **Preferred for local development** — authoring and calibrating the case in front of you: drop the JSON into `data/workflows/`, `--filter` it, iterate. Also the only home of the `agents` tier and of a `replay`-seeded case (reconstructed from a trace at run time, so no suite can hold it); since the corpus migration the directory holds only those, not the full suite. |80| **`langtracer`** (`--source langtracer --suite baseline`) | Bigger runs (the full corpus or a whole tier), re-running specific cases that already live in the suite, and CI — which always runs this way. Needs `LANGTRACER_URL`/`LANGTRACER_API_KEY` in your env. |81 82## Configuration & secrets83 84The harness reads its configuration from environment variables — how you supply85them is up to you. A common setup is a gitignored local env file loaded with86`dotenvx` (`.env.eval.example` is a starting template), but any env mechanism87works. The pieces, regardless of how they're loaded:88 89- the **model key** the eval helper uses (mock-gen / verifier / user-proxy /90  judge);91- the **login** the eval signs in with;92- optionally `LANGSMITH_API_KEY` to record experiments and compare against a93  baseline (note: setting this locally also writes your run into the shared94  dataset — leave it unset for throwaway exploration);95- optionally `CONTEXT7_API_KEY` to improve mock realism for less-common services.96 97Describe-what-you-need, not a fixed recipe: on a dev instance you already use,98the login and model are usually already set, and you only add the eval-specific99keys.100 101## Run locally against a dev instance102 103The pieces above assume a configured instance; this is the concrete recipe that104works end-to-end (direct Daytona mode, no proxy), plus the footguns that don't105surface until you hit them.106 107**Point the run with the `--base-url` flag, not the `N8N_EVAL_BASE_URL` env var.**108If you load env via `dotenvx` / `.env.local` (a common dev setup), the file's109value silently overrides the `N8N_EVAL_BASE_URL` you export, and the run110authenticates against the wrong instance (a confusing 401). The CLI `--base-url`111flag wins — use it.112 113**Direct-mode env combo** (bypass the proxy; your own Daytona + Anthropic keys):114 115- `N8N_AI_ASSISTANT_BASE_URL=` **empty** — direct mode is only selected when the116  proxy base URL is unset.117- `DAYTONA_API_KEY` + `DAYTONA_API_URL` — sandbox auth; without them every build118  crashes at `DaytonaAuthManager requires exactly one of staticApiKey or119  getAuthToken`.120- `ANTHROPIC_API_KEY` — the non-proxy orchestrator reads this (not just121  `N8N_AI_ANTHROPIC_KEY`); the eval helper (mock-gen / verify / judge) reads122  either. **You usually don't need a separate key: `.env.eval` already carries123  `N8N_AI_ANTHROPIC_KEY`** — mirror it rather than hunting for another one124  (`ANTHROPIC_API_KEY="$(grep '^N8N_AI_ANTHROPIC_KEY=' .env.eval | cut -d= -f2-)"`,125  or just export it inside the `dotenvx` child). Check the file before concluding126  a key is missing — and grep its names with `^[A-Za-z0-9_]+=`, since a127  `^[A-Z_]*=` pattern silently drops every `N8N_*` var (the digit).128- `E2E_TESTS=true` — exposes `POST /rest/e2e/reset` so you can seed a known owner.129 130**Seed an owner** (a fresh instance has none → login 401s). Full payload shape in131the [README](../../../packages/@n8n/instance-ai/evaluations/README.md) quick132start:133 134```bash135curl -sf -X POST <base>/rest/e2e/reset -H 'Content-Type: application/json' \136  -d '{"owner":{"email":"nathan@n8n.io","password":"PlaywrightTest123","firstName":"Eval","lastName":"Owner"},"admin":{"email":"admin@n8n.io","password":"PlaywrightTest123","firstName":"Admin","lastName":"User"},"members":[],"chat":{"email":"chat@n8n.io","password":"PlaywrightTest123","firstName":"Chat","lastName":"User"}}'137```138 139**Run isolated alongside an already-running dev instance** (rather than killing140it): a running instance holds both its main port *and* the task-broker port141(default 5679), and the default DB is one shared SQLite file in `~/.n8n`. Give the142second instance its own everything:143 144```bash145# .env.eval alone is enough — it carries N8N_AI_*, the sandbox/Daytona keys and146# N8N_AI_ANTHROPIC_KEY. Skip .env.local: its staging base-url / port-5678147# defaults fight the settings below.148N8N_PORT=5680 N8N_RUNNERS_BROKER_PORT=5681 N8N_USER_FOLDER=/tmp/n8n-eval-run \149E2E_TESTS=true N8N_AI_ASSISTANT_BASE_URL= \150  npx dotenvx run -f .env.eval -- \151  sh -c 'ANTHROPIC_API_KEY="$N8N_AI_ANTHROPIC_KEY" exec pnpm start'152# then seed the owner (above), and run the eval with: --base-url http://localhost:5680153```154 155**Don't reach for `DB_SQLITE_POOL_SIZE=0`** if you find it in an older note. It156can't disable pooling: the schema is `.int().gte(1)`, so `0` is rejected and157`@Env` warns (`Invalid value for DB_SQLITE_POOL_SIZE … Falling back to default158value.`) and keeps the default of 3. There is no non-pooled path to select159anyway — `getSqliteConnectionOptions()` always returns `type: 'sqlite-pooled'`.160`POST /rest/e2e/reset` seeds the owner fine under the pooled driver.161 162`pnpm start` (built dist) is enough — no need for `pnpm dev:ai`. Case JSON is read163from source at run time, so new/edited cases need no rebuild.164 165**Two WARNs are benign, not failures:** `Run debug capture skipped … Run debug is166not enabled` (a 404 from an optional debug endpoint) and workflow-checks167`errored, excluded from scoring` — both are expected locally and don't affect168your case's pass/fail.169 170## Parallel lanes171 172The **build** is the slow step and is capped at **4 concurrent builds per173instance**, so throughput scales with the number of instances, not just174`--concurrency`.175 176**Watch for false timeouts under contention:** a batch larger than the cap can177queue a *healthy* case behind that limit until it hits the per-iteration timeout178and reports `BUILD FAILED: Run timed out` — a run-capacity artifact, not a case179defect (a case that builds in ~3 min solo can "time out" at 900s in a crowded180batch). Re-run the suspect solo (`--concurrency 1`) to confirm it builds in time,181and for batches beyond ~a dozen cases fan out across lanes rather than just182raising `--concurrency` on one instance. Two ways to fan out:183 184- **`scripts/run-eval-lanes.sh`** spins up N lanes as **docker containers**,185  seeds a user on each, and runs the eval with the base-URLs wired together.186  It needs the **local image built first** (`INCLUDE_TEST_CONTROLLER=true pnpm187  build:docker`); pass `--build` to (re)build it, which you must do after any188  code change (the image is a snapshot). Extra eval args pass through after `--`.189 190  ```bash191  # from packages/@n8n/instance-ai/192  ./scripts/run-eval-lanes.sh --instance-count 5 --tier pr193  ./scripts/run-eval-lanes.sh --instance-count 3 --build -- --filter contact-form194  ```195 196- **Comma-separated `--base-url`** to fan across instances you're already197  running; a work-stealing allocator dispatches each build to a free lane.198 199  ```bash200  pnpm eval:instance-ai --base-url http://localhost:5678,http://localhost:6678201  ```202 203## Tiers204 205Each case declares a `datasets` array (default `["full"]`) — free-form logical206groupings, propagated to LangSmith as example splits so `--tier <name>` maps to a207server-side filter. The two that matter for CI:208 209- **`full`** — every case; nightly / full-suite runs.210- **`pr`** — curated thin set for the PR gate: high baseline reliability +211  capability diversity.212 213Other values group cases logically (e.g. `behaviour` for conversation-behaviour214cases, `seeded` for transient `replay` cases kept out of CI). For a new215local case, put the value in its `datasets` array before pushing; for a case216already in LangTracer, edit `datasets` there — `eval:langtracer-push`217deliberately does not re-sync tier-only edits to an existing case. **Only218promote to `pr` after `--iterations 5+` shows it's reliably green** — a flaky219case in the gate poisons it.220 221## Baselines & regression222 223When `LANGSMITH_API_KEY` is set, every run auto-compares against the most recent224experiment named `instance-ai-baseline-*` and writes `eval-pr-comment.md`.225Regression tiers are computed statistically (`comparison/statistics.ts`) so226small-N PR runs don't flag noise. Comparison is best-effort — it never fails a227run.228 229**Refresh the baseline** explicitly (no auto-refresh), on `master`, with high N230for low noise:231 232```bash233# with your env loaded and LANGSMITH_API_KEY set, from packages/@n8n/instance-ai/234# (--dataset/--baseline-prefix mirror CI's pins — langtracer mode otherwise235# derives suite-scoped names and later runs would never find this baseline)236pnpm eval:instance-ai --source langtracer --suite baseline \237  --dataset instance-ai-workflow-evals --baseline-prefix instance-ai-baseline- \238  --experiment-name instance-ai-baseline --iterations 10239```240 241LangSmith appends a random suffix; the most-recently-started `instance-ai-baseline-*`242becomes the next comparison target. In CI, the same is a workflow dispatch:243 244```bash245gh workflow run test-evals-instance-ai.yml -f experiment-name=instance-ai-baseline -f iterations=10246gh workflow run ci-instance-ai-evals.yml -f pr=<number>   # re-run evals against a PR's head247```248 249An isolated cohort (e.g. MCP) must override **both** `--dataset` and250`--baseline-prefix` — overriding one still touches shared Instance AI data (the251CLI warns).252 253## Reading the result254 255The `workflow-eval-report.html` in the run's `.data/` dir is the fastest256debugger — full transcript, per-node traces, intercepted requests + mock257responses, and the verifier's reasoning. `eval-results.json` is the258machine-readable companion. (See the SKILL's "Outputs of a run".)259