n8n:create-agent-builder-eval

Source excerpt starting at line 1.

1# Local Agent Builder eval setup2 3## Required LangTracer access4 5Generate a key on the6[LangTracer API keys page](https://lang-tracer.n8n-maintenance.workers.dev/account?section=api).7Add it to `.env.local` at the repository root:8 9```env10LANGTRACER_URL=https://lang-tracer.n8n-maintenance.workers.dev11LANGTRACER_API_KEY=<generated-key>12```13 14Do not paste the key into chat or put it in a command. The main skill checks15that both variables exist before it creates an eval.16 17## Remaining environment18 19Create `.env.local` at the repository root. Use one Instance AI model provider20and one sandbox provider.21 22```env23N8N_INSTANCE_AI_MODEL=anthropic/claude-sonnet-4-624N8N_INSTANCE_AI_MODEL_API_KEY=<model-key>25ANTHROPIC_API_KEY=<same-model-key>26N8N_AI_ASSISTANT_BASE_URL=27 28N8N_INSTANCE_AI_SANDBOX_ENABLED=true29N8N_INSTANCE_AI_SANDBOX_PROVIDER=daytona30DAYTONA_API_URL=https://app.daytona.io/api31DAYTONA_API_KEY=<daytona-key>32 33N8N_EVAL_EMAIL=nathan@n8n.io34N8N_EVAL_PASSWORD=PlaywrightTest12335```36 37Use the n8n sandbox service instead of Daytona when it is available. Set38`N8N_INSTANCE_AI_SANDBOX_PROVIDER=n8n-sandbox`,39`N8N_SANDBOX_SERVICE_URL`, and, when required,40`N8N_SANDBOX_SERVICE_API_KEY`.41 42LangSmith is optional. Unset `LANGSMITH_API_KEY` and `LANGSMITH_TRACING` for a43throwaway direct run. This keeps the run out of the shared LangSmith dataset.44 45## Build after code changes46 47From the repository root:48 49```bash50(51  pnpm --filter n8n... build > build.log 2>&152  build_status=$?53  tail -n 20 build.log54  exit "$build_status"55)56```57 58Restart the instance after each build. A running process keeps the old code in59memory.60 61## Start an isolated instance62 63Create a temporary user directory:64 65```bash66mktemp -d /tmp/n8n-agent-eval.XXXXXX67```68 69Use the printed path as `N8N_USER_FOLDER`. Start the built CLI from70`packages/cli`:71 72```bash73N8N_PORT=5680 \74N8N_RUNNERS_BROKER_PORT=5681 \75N8N_USER_FOLDER=<printed-temp-directory> \76E2E_TESTS=true \77N8N_ENABLED_MODULES=instance-ai,agents \78N8N_AI_ENABLED=true \79pnpm exec dotenvx run -f ../../.env.local -- pnpm start80```81 82Use different ports when 5680 or 5681 is busy.83 84## Seed the owner85 86Run this once after the isolated instance starts:87 88```bash89curl -sf -X POST http://localhost:5680/rest/e2e/reset \90  -H 'Content-Type: application/json' \91  -d '{"owner":{"email":"nathan@n8n.io","password":"PlaywrightTest123","firstName":"Eval","lastName":"Owner"},"admin":{"email":"admin@n8n.io","password":"PlaywrightTest123","firstName":"Admin","lastName":"User"},"members":[],"chat":{"email":"chat@n8n.io","password":"PlaywrightTest123","firstName":"Chat","lastName":"User"}}'92```93 94## Run a local disk case95 96From `packages/@n8n/instance-ai`:97 98```bash99pnpm exec dotenvx run -f ../../../.env.local -- \100  env -u LANGSMITH_API_KEY -u LANGSMITH_TRACING \101  pnpm eval:instance-ai \102  --base-url http://localhost:5680 \103  --filter <slug> \104  --tier agents \105  --concurrency 1 \106  --verbose107```108 109The run writes `eval-results.json`, an HTML report, and a PR-comment preview.110The warnings about disabled run-debug capture are benign.111