evals/evals.json
4,904 tokens · o200k_base · 22,803 bytes
Source excerpt starting at line 1.{ "skill_name": "astro-code-review", "evals": [ { "id": 1, "prompt": "Review this self-contained historical PR snapshot as the complete scope. The eval workspace intentionally has no checkout or network context; use only the supplied evidence and do not fetch or inspect current repository policy.\n\nPR: https://github.com/withastro/astro/pull/17546\nScope: `bbc1ec9715160e25eb6a6fee2e133386414c0c00..d605159187f3cebe7812b14ac3cb31dfb7faf1cb`\n\nThe PR description establishes these requirements:\n- Return reviews as raw Markdown that can be copied elsewhere.\n- Do not report dependent-package version bumps that Changesets generates during release versioning.\n- Check that pending changesets cover every directly modified package that qualifies under the repository's historical changeset policy.\n- Fix the YAML frontmatter parsing bug that prevents the triage bot from loading skill metadata.\n\nA maintainer review established one additional scope requirement: the skill description must target pull-request review and must not cause the triage bot to activate this skill while investigating ordinary bug reports. The author replied that this was addressed in `d605159`.\n\nHistorical repository policy for this snapshot is authoritative: changes confined to `.agents/` do not require a changeset.\n\nComplete patch:\n\n```diff\ndiff --git a/.agents/skills/astro-code-review/SKILL.md b/.agents/skills/astro-code-review/SKILL.md\n--- a/.agents/skills/astro-code-review/SKILL.md\n+++ b/.agents/skills/astro-code-review/SKILL.md\n@@ -1,6 +1,6 @@\n ---\n name: astro-code-review\n-description: Perform a static, read-only code review of an Astro feature, bug fix, branch, commit, diff, or local working tree. Use this skill whenever the user asks to review local changes, review their current branch, self-review work before a pull request, or check a feature or fix for correctness, security, tests, simplicity, runtime portability, error handling, comments, behavior documentation, and changeset coverage. This skill reports findings only: it never edits code or runs project code, tests, builds, checks, or scripts.\n+description: \"Perform a static, read-only code review of an Astro pull request or of a local branch, commit range, diff, patch, or working tree being prepared as a pull request. Use this skill only when the user asks to review PR changes, review their current branch before opening or updating a PR, or self-review a PR for correctness, security, tests, simplicity, runtime portability, error handling, comments, behavior documentation, and changeset coverage. Do not use it for issue review, bug-report investigation, reproduction, diagnosis, or triage. This skill reports findings only: it never edits code or runs project code, tests, builds, checks, or scripts\"\n ---\n \n # Astro Code Review\n@@ -210,6 +210,8 @@ When package dependencies change:\n - A package's TypeScript build references should mirror its workspace dependencies.\n - A new package needs the standard build/test solution files and a reference from the root `tsconfig.json`.\n \n+Do not inspect or report missing dependent-package version bumps or internal dependency-range updates that arise solely because another workspace package is being released. Changesets computes these propagation updates from [`.changeset/config.json`](../../../.changeset/config.json) when release versions are applied, so feature pull requests should not make them manually. Continue to check dependency declarations, workspace protocols, runtime compatibility, TypeScript references, and directly modified packages against the changeset criteria below.\n+\n Use the \"TypeScript project references\" section in [`CONTRIBUTING.md`](../../../CONTRIBUTING.md) and the package-boundary guidance in [`astro-developer/constraints.md`](../astro-developer/constraints.md).\n \n #### Generated Files and Feature Registries\n@@ -297,10 +299,11 @@ Keep comment findings non-invasive. Report inaccurate or misleading documentatio\n \n ### Changeset Coverage\n \n-Follow the current repository policy in [`CONTRIBUTING.md`](../../../CONTRIBUTING.md) and the [`changeset` skill](../changeset/SKILL.md): every pull request that modifies a package under `packages/` requires a pending changeset. Changes outside packages, including `examples/`, do not require one by themselves.\n+Require a pending changeset for each package whose non-test source is directly modified when the change affects its user-facing API, including observable API behavior. Test-only changes and changes outside packages, including `examples/`, do not require one by themselves. Follow the current repository policy in [`CONTRIBUTING.md`](../../../CONTRIBUTING.md) and use the [`changeset` skill](../changeset/SKILL.md) as the format reference.\n \n - Check for a newly added `.changeset/*.md` file in the review scope. Configuration, README, and prerelease metadata files are not pending changesets.\n-- If packages changed, confirm the changeset frontmatter covers the affected package names.\n+- For each qualifying source change, confirm the changeset frontmatter names the package containing that source.\n+- Do not require a package in changeset frontmatter solely because it depends on another package receiving a bump; Changesets handles that dependent bump during release versioning.\n - Report a missing or ineffective changeset as a finding.\n - Do not invoke the changeset skill or create or edit a changeset.\n \n@@ -320,6 +323,8 @@ Report only issues that are actionable and supported by the inspected code.\n \n ## Report Format\n \n+Return the complete review as raw, unrendered Markdown inside a single fenced code block. Put no text before or after the code block, and do not escape Markdown syntax inside it. The code block's contents must be directly pasteable into a GitHub comment.\n+\n Put findings first, order required findings by severity, and list optional improvements last. Do not lead with a summary or praise.\n \n Use exactly one severity and one primary category per finding. Required findings use `high`, `medium`, or `low`; non-blocking improvements use `optional`.\n@@ -331,6 +336,10 @@ Use exactly one severity and one primary category per finding. Required findings\n \n Use one of these categories: `design`, `correctness`, `security`, `runtime`, `completeness`, `error-handling`, `tests`, `maintainability`, `documentation`, or `changeset`. Choose the category that describes the root cause rather than a downstream symptom. `completeness` covers missing API, configuration, export, dependency, generated-file, or registry wiring.\n \n+Use this structure:\n+\n+````md\n+```\n ## Findings\n \n - `[high][security]` `path/to/file.ts:42` - Short title. Explain the triggering scenario, impact, and minimal remediation direction.\n@@ -348,10 +357,9 @@ Scope: `<base>` through the current working tree, plus listed untracked files.\n Changeset: present and covers `<packages>` | missing for `<packages>` | not required.\n Validation: Static review only; no project code, tests, builds, or checks were run.\n Fetch: updated `origin/main` | fetch failed and local `origin/main` was used | not needed for the supplied scope.\n-\n ```\n+````\n \n Severity reflects impact, not confidence.\n \n If there are no required or optional findings, write `No findings.` under `## Findings`. Still include review status and mention any residual uncertainty caused by unavailable requirements or context.\n-```\n```", "expected_output": "A no-findings review in the skill's exact fenced-Markdown format. It traces all four PR-description requirements and the resolved trigger-scope requirement to the final patch, treats the `.agents`-only change as not requiring a changeset, and concludes that the complete supplied scope is ready to merge based on static review.", "files": [], "assertions": [ "The response consists solely of one fenced code block containing unescaped Markdown, with no text before or after it.", "The `## Findings` section contains exactly `No findings.` and the response contains no optional suggestion or invented concern.", "The merge-requirement assessment identifies the raw-Markdown report, generated dependent-bump suppression, qualifying-package changeset coverage, and valid frontmatter as satisfied.", "The merge-requirement assessment recognizes the maintainer's resolved scope decision and explains that the final description targets PR review while excluding bug investigation and triage.", "The review status uses `ready to merge based on static review`, identifies the exact supplied PR range, reports `Changeset: not required`, and says GitHub context was supplied only.", "No files are changed and no shell command, network request, test, build, check, script, or project command is run or claimed to have been run." ] }, { "id": 2, "prompt": "Review this self-contained historical PR snapshot as the complete scope. The eval workspace intentionally has no checkout or network context; use only the supplied evidence and do not fetch or inspect current repository policy.\n\nPR: https://github.com/withastro/astro/pull/17546\nScope: `bbc1ec9715160e25eb6a6fee2e133386414c0c00..d381b9160c998eb11caf570cf430e901da55292d`\n\nThe PR description establishes these requirements:\n- Return reviews as raw Markdown that can be copied elsewhere.\n- Do not report dependent-package version bumps that Changesets generates during release versioning.\n- Check that pending changesets cover every directly modified package that qualifies under the repository's historical changeset policy.\n- Fix the YAML frontmatter parsing bug that prevents the triage bot from loading skill metadata.\n\nAt this head, a maintainer left an unresolved review requirement on the changed description: the skill must target pull-request review and must not cause the triage bot to activate it while investigating ordinary bug reports. The maintainer specifically identified the broad `feature, bug fix` and `whenever the user asks` trigger language as the cause of false-positive activation.\n\nHistorical repository policy for this snapshot is authoritative: changes confined to `.agents/` do not require a changeset.\n\nComplete patch:\n\n```diff\ndiff --git a/.agents/skills/astro-code-review/SKILL.md b/.agents/skills/astro-code-review/SKILL.md\n--- a/.agents/skills/astro-code-review/SKILL.md\n+++ b/.agents/skills/astro-code-review/SKILL.md\n@@ -1,6 +1,6 @@\n ---\n name: astro-code-review\n-description: Perform a static, read-only code review of an Astro feature, bug fix, branch, commit, diff, or local working tree. Use this skill whenever the user asks to review local changes, review their current branch, self-review work before a pull request, or check a feature or fix for correctness, security, tests, simplicity, runtime portability, error handling, comments, behavior documentation, and changeset coverage. This skill reports findings only: it never edits code or runs project code, tests, builds, checks, or scripts.\n+description: \"Perform a static, read-only code review of an Astro feature, bug fix, branch, commit, diff, or local working tree. Use this skill whenever the user asks to review local changes, review their current branch, self-review work before a pull request, or check a feature or fix for correctness, security, tests, simplicity, runtime portability, error handling, comments, behavior documentation, and changeset coverage. This skill reports findings only: it never edits code or runs project code, tests, builds, checks, or scripts\"\n ---\n \n # Astro Code Review\n@@ -210,6 +210,8 @@ When package dependencies change:\n - A package's TypeScript build references should mirror its workspace dependencies.\n - A new package needs the standard build/test solution files and a reference from the root `tsconfig.json`.\n \n+Do not inspect or report missing dependent-package version bumps or internal dependency-range updates that arise solely because another workspace package is being released. Changesets computes these propagation updates from [`.changeset/config.json`](../../../.changeset/config.json) when release versions are applied, so feature pull requests should not make them manually. Continue to check dependency declarations, workspace protocols, runtime compatibility, TypeScript references, and directly modified packages against the changeset criteria below.\n+\n Use the \"TypeScript project references\" section in [`CONTRIBUTING.md`](../../../CONTRIBUTING.md) and the package-boundary guidance in [`astro-developer/constraints.md`](../astro-developer/constraints.md).\n \n #### Generated Files and Feature Registries\n@@ -297,10 +299,11 @@ Keep comment findings non-invasive. Report inaccurate or misleading documentatio\n \n ### Changeset Coverage\n \n-Follow the current repository policy in [`CONTRIBUTING.md`](../../../CONTRIBUTING.md) and the [`changeset` skill](../changeset/SKILL.md): every pull request that modifies a package under `packages/` requires a pending changeset. Changes outside packages, including `examples/`, do not require one by themselves.\n+Require a pending changeset for each package whose non-test source is directly modified when the change affects its user-facing API, including observable API behavior. Test-only changes and changes outside packages, including `examples/`, do not require one by themselves. Follow the current repository policy in [`CONTRIBUTING.md`](../../../CONTRIBUTING.md) and use the [`changeset` skill](../changeset/SKILL.md) as the format reference.\n \n - Check for a newly added `.changeset/*.md` file in the review scope. Configuration, README, and prerelease metadata files are not pending changesets.\n-- If packages changed, confirm the changeset frontmatter covers the affected package names.\n+- For each qualifying source change, confirm the changeset frontmatter names the package containing that source.\n+- Do not require a package in changeset frontmatter solely because it depends on another package receiving a bump; Changesets handles that dependent bump during release versioning.\n - Report a missing or ineffective changeset as a finding.\n - Do not invoke the changeset skill or create or edit a changeset.\n \n@@ -320,6 +323,8 @@ Report only issues that are actionable and supported by the inspected code.\n \n ## Report Format\n \n+Return the complete review as raw, unrendered Markdown inside a single fenced code block. Put no text before or after the code block, and do not escape Markdown syntax inside it. The code block's contents must be directly pasteable into a GitHub comment.\n+\n Put findings first, order required findings by severity, and list optional improvements last. Do not lead with a summary or praise.\n \n Use exactly one severity and one primary category per finding. Required findings use `high`, `medium`, or `low`; non-blocking improvements use `optional`.\n@@ -331,6 +336,10 @@ Use exactly one severity and one primary category per finding. Required findings\n \n Use one of these categories: `design`, `correctness`, `security`, `runtime`, `completeness`, `error-handling`, `tests`, `maintainability`, `documentation`, or `changeset`. Choose the category that describes the root cause rather than a downstream symptom. `completeness` covers missing API, configuration, export, dependency, generated-file, or registry wiring.\n \n+Use this structure:\n+\n+````md\n+```\n ## Findings\n \n - `[high][security]` `path/to/file.ts:42` - Short title. Explain the triggering scenario, impact, and minimal remediation direction.\n@@ -348,10 +357,9 @@ Scope: `<base>` through the current working tree, plus listed untracked files.\n Changeset: present and covers `<packages>` | missing for `<packages>` | not required.\n Validation: Static review only; no project code, tests, builds, or checks were run.\n Fetch: updated `origin/main` | fetch failed and local `origin/main` was used | not needed for the supplied scope.\n-\n ```\n+````\n \n Severity reflects impact, not confidence.\n \n If there are no required or optional findings, write `No findings.` under `## Findings`. Still include review status and mention any residual uncertainty caused by unavailable requirements or context.\n-```\n```", "expected_output": "A concise findings-first review in the required fenced format. It reports exactly one merge-relevant requirement gap at the changed description: the broad trigger still activates the skill for ordinary bug investigation and therefore has not satisfied the maintainer's explicit PR-review-only scope decision. It recognizes the four original requirements as satisfied, adds no optional padding, and concludes that changes are required.", "files": [], "assertions": [ "The response consists solely of one fenced code block containing the required Markdown review sections.", "The findings contain exactly one issue, located at `.agents/skills/astro-code-review/SKILL.md:3` and categorized as a merge-relevant requirements gap.", "The finding explains that the broad feature and bug-fix trigger can make the triage bot load the review skill outside pull-request review, and directs narrowing the description to PR review while excluding bug investigation or triage.", "The merge-requirement assessment recognizes the raw-Markdown report, generated dependent-bump suppression, qualifying-package changeset coverage, and valid frontmatter as satisfied while marking the trigger-scope requirement as unmet.", "The response contains no unrelated finding, optional suggestion, style preference, or request for more tests.", "The review status uses `changes required`, identifies the exact supplied PR range, reports `Changeset: not required`, and says GitHub context was supplied only.", "No files are changed and no shell command, network request, test, build, check, script, or project command is run or claimed to have been run." ] }, { "id": 3, "prompt": "Review only the supplied review-context excerpt from historical PR https://github.com/withastro/astro/pull/17526 at `bcf733791f55b7b445e94f1bb6be7349ab3c65b7`. This is intentionally a partial scope for deciding whether the quoted feedback leaves a pre-merge blocker. Do not inspect the checkout, use network access, or issue a whole-PR merge verdict.\n\nPR purpose relevant to this excerpt: add an Astro code-review skill that catches easy-to-spot bugs, checks test coverage and codebase guidelines, and detects runtime portability issues such as Node.js API usage.\n\nReview decisions:\n- A maintainer requested a concise section checking common security vulnerabilities such as XSS and CSRF. The author implemented it, replied `Done!`, and the maintainer later approved the head. This is an accepted merge requirement.\n- A maintainer asked why shell pipelines and scripts were prohibited. The author explained that they may require unavailable permissions or executables. The maintainer closed the discussion with `Nah, it's not a big deal at this stage, just something I wondered about.` This is explicitly dismissed feedback, not a merge requirement.\n- Changes in files outside this excerpt are not available and must not be assessed.\n\nRelevant current-head excerpt:\n\n```diff\ndiff --git a/.agents/skills/astro-code-review/SKILL.md b/.agents/skills/astro-code-review/SKILL.md\nnew file mode 100644\n--- /dev/null\n+++ b/.agents/skills/astro-code-review/SKILL.md\n@@ -13,6 +13,10 @@ This is a static review. Preserve the developer's working tree exactly as found.\n+- Do not run project code, package-manager commands, scripts, tests, type checks, linters, formatters, builds, benchmarks, dev servers, or browser automation.\n+\n+Use file-reading, globbing, and text-search tools for all other investigation. Do not use shell pipelines or scripts to analyze source code.\n+\n@@ -97,6 +101,7 @@ Review in this order so correctness and compatibility are not displaced by style suggestions:\n 1. Design and architectural layer placement\n 2. Functional correctness and regressions\n+3. Security and trust boundaries\n 4. Runtime portability, state ownership, and generated output\n@@ -137,6 +142,15 @@ Do not report a theoretical edge case without explaining how the changed code can encounter it and what fails.\n+### Security and Trust Boundaries\n+\n+Apply security review when changed code accepts less-trusted input, emits executable or interpreted output, handles credentials or secrets, changes a public request endpoint, or modifies an existing defense. Trace a reachable input to its sink or protection boundary and explain the attacker capability and impact. Do not report a vulnerability from a dangerous-looking name or API alone.\n+\n+Before reviewing a security-sensitive change, use read-only documentation search or fetch tools rather than model memory. Consult current XSS guidance for rendering or DOM-insertion changes and current CSRF guidance for state-changing request paths. Establish Astro's public security contract from applicable documentation and the checked-out implementation.\n+\n+When a change reaches an Astro-owned security mechanism, establish its contract from the current implementation, neighboring tests, and public documentation. Verify that the change preserves that contract and its trust boundary.\n```", "expected_output": "A no-findings review of the supplied partial scope in the exact fenced-Markdown format. It marks the requested security guidance as satisfied, treats the pipelines-and-scripts exchange as dismissed non-blocking feedback, omits optional commentary, and uses the partial-scope verdict rather than claiming that the entire historical PR is merge-ready.", "files": [], "assertions": [ "The response consists solely of one fenced code block containing unescaped Markdown.", "The `## Findings` section contains exactly `No findings.` and there is no `## Suggestions` section.", "The merge-requirement assessment identifies the requested XSS and CSRF security-review guidance as satisfied by the current-head excerpt.", "The response does not turn the explicitly dismissed pipelines-and-scripts discussion into a finding, question, suggestion, or unmet requirement.", "The review status uses `no blockers found in supplied scope`, limits the scope to the supplied PR excerpt, reports the changeset as not assessable from supplied scope, and discloses that no whole-PR verdict was made.", "No files are changed and no shell command, network request, test, build, check, script, or project command is run or claimed to have been run." ] } ]}