n8n:create-agent-builder-eval

Source excerpt starting at line 1.

1---2name: n8n:create-agent-builder-eval3description: >-4  Authors and calibrates Instance AI evaluations that build standalone n8n5  Agents through Agent Builder. Use when a change under6  packages/cli/src/modules/agents affects build-agent routing, Agent setup,7  model or credential selection, tools, MCP servers, integrations, skills,8  tasks, testing, or user-facing build responses. Requires LangTracer access9  before authoring so each finished case can be published.10---11 12# Create an Agent Builder eval13 14Use the shared Instance AI eval harness. Agent cases use an Agent-specific15authoring directory, dataset, and LangTracer suite.16 17## Required LangTracer preflight18 19Run this check before sourcing, drafting, or writing an eval. Run it from20`packages/@n8n/instance-ai`:21 22```bash23pnpm exec dotenvx run -f ../../../.env.local -- \24  sh -c 'test -n "${LANGTRACER_URL:-}" && test -n "${LANGTRACER_API_KEY:-}"'25```26 27If the check fails, stop before creating an eval file. Ask the user to:28 291. Generate a key on the30   [LangTracer API page](https://lang-tracer.n8n-maintenance.workers.dev/account?section=api).312. Add these variables to the repository root `.env.local` file:32 33   ```env34   LANGTRACER_URL=https://lang-tracer.n8n-maintenance.workers.dev35   LANGTRACER_API_KEY=<generated-key>36   ```37 383. Confirm when the environment is ready.39 40Do not ask the user to paste the key into chat. Do not print or inspect its41value. Rerun the check after the user confirms. Continue only when it passes.42 43## Non-negotiable routing44 45- Author the case at46  `packages/@n8n/instance-ai/evaluations/data/agents/<slug>.json`.47- Set `"datasets": ["agents"]`.48- Push general Agent Builder cases to49  [Instance AI capabilities — agents](https://lang-tracer.n8n-maintenance.workers.dev/suites/10).50  Its suite slug is `agents`.51- Do not commit the case JSON. LangTracer is the durable source of truth.52- Commit changes to this skill, the harness, and CI when applicable.53 54The disk runner loads both `data/agents/` and `data/workflows/`. A misplaced55Agent case can therefore pass locally. That does not make the location correct.56 57## Decide what the case proves58 59Write the smallest user request that exercises the changed behavior.60 61- Use `processExpectations` for the Instance AI conversation and final response.62- Use `outcomeExpectations` for the created Agent artifact and its configuration.63- Use `executionScenarios` only when the built Agent must run to prove the behavior.64- Declare only the credentials that the build must see.65 66The harness captures the Agent configuration and authored skills. It supplies67them to the expectation judge. A scenario-less Agent case is valid when process68or outcome expectations can prove the behavior.69 70Use the substitution test for every expectation. A correct alternative build71must pass. A build that misses the requested behavior must fail.72 73For multi-turn, seeded, or capability-gap cases, follow the case-shape and74calibration rules in [create-instance-ai-eval](../create-instance-ai-eval/SKILL.md).75This skill overrides its workflow directory and suite guidance for Agent cases.76 77## Draft the case78 79Start with this shape:80 81```json82{83  "description": "The Agent Builder behavior this case guards.",84  "conversation": [85    { "role": "user", "text": "Build me an Agent that ..." }86  ],87  "complexity": "simple",88  "tags": ["agent", "agent-build", "<capability>"],89  "credentials": [{ "type": "<credentialType>", "name": "<display name>" }],90  "processExpectations": [91    "The final response ..."92  ],93  "outcomeExpectations": [94    "A standalone Agent was created and no workflow was created.",95    "The Agent ..."96  ],97  "datasets": ["agents"]98}99```100 101Keep the prompt in the user's voice. Do not tell Instance AI which internal102tools or configuration fields to use unless that choice is the behavior under103test.104 105## Validate and run locally106 107Read [local-setup.md](local-setup.md) when the machine does not already have an108eval instance and environment file.109 110From `packages/@n8n/instance-ai`:111 112```bash113pnpm exec tsx -e "import { loadAgentEvalTestCasesWithFiles } from './evaluations/data/agents/index.ts'; const matches = loadAgentEvalTestCasesWithFiles('<slug>'); if (matches.length !== 1) throw new Error('Expected exactly one Agent eval case, found ' + matches.length); console.log(matches[0].fileSlug)"114 115pnpm eval:instance-ai \116  --base-url http://localhost:5680 \117  --filter <slug> \118  --tier agents \119  --concurrency 1 \120  --keep-workflows \121  --verbose122```123 124Use `eval:instance-ai` for a new disk case. `eval:agents` reads the published125LangTracer suite and is for running cases that are already there.126 127Inspect the transcript, the rendered Agent artifact, and each judge reason.128Do not accept a green result when a conditional expectation never occurred.129Do not weaken an expectation to hide a real Agent Builder defect.130 131## CI coverage132 133The Instance AI PR gate checks the files changed by the PR. A change under134`packages/cli/src/modules/agents/` selects the `Instance AI capabilities — agents`135suite through its `agents` slug. It also selects the `agents` dataset and uses136an absolute pass gate. The run uses a suite-scoped LangSmith cohort. It does not137write to the workflow dataset or compare against the workflow baseline. Other138Instance AI changes select the `baseline` suite and its `pr` dataset.139 140The gate runs when a PR opens, reopens, or becomes ready for review. It does not141run for each new push. Use the PR gate's manual dispatch after a later push.142 143## Credential behavior144 145Declared credentials are real n8n credential records with placeholder data.146The eval thread limits the builder to those credential IDs.147 148Agent Builder model catalog requests return deterministic fake models during an149eval. They do not decrypt the placeholder model credential or call its provider.150Production model catalog requests remain live.151 152This mock covers catalog lookup only. A builder `call_agent` action and an Agent153`executionScenario` run the target Agent model. They need a working provider154credential such as `EVAL_OPENAI_API_KEY`. A build-only case does not need one.155 156## Calibrate and publish157 1581. Run the case once with `--concurrency 1`.1592. Confirm that Instance AI called `build-agent`.1603. Confirm that the expected Agent artifact was captured.1614. Classify each red as a product gap, harness limitation, or non-determinism.1625. Use `--iterations 5` before adding a case to a gating tier.1636. Preview the LangTracer change.1647. Push it to `agents`.165 166```bash167pnpm exec dotenvx run -f ../../../.env.local -- \168  pnpm eval:langtracer-push --suite agents --dry-run --changed169 170pnpm exec dotenvx run -f ../../../.env.local -- \171  pnpm eval:langtracer-push --suite agents --changed172```173 174The push needs `LANGTRACER_URL` and `LANGTRACER_API_KEY`. Generate a key on the175[LangTracer API page](https://lang-tracer.n8n-maintenance.workers.dev/account?section=api).176Report the case and suite as clickable LangTracer links. Delete the local JSON177after a successful push.178 179## Completion checklist180 181- The LangTracer environment preflight passed before authoring.182- The case is in `data/agents/` and uses the `agents` dataset.183- The strict loader accepts it.184- A local run captures a standalone Agent.185- The transcript proves each process expectation was exercised.186- The rendered Agent artifact proves each outcome expectation.187- Repeated runs are stable enough for the selected tier.188- The case is pushed to `Instance AI capabilities — agents` with suite slug `agents`.189- Agent Builder PR changes select the `agents` suite in CI.190- The local case JSON is not committed.191