differential-fuzzer

Source excerpt starting at line 1.

1---2name: differential-fuzzer3description: Information about the differential fuzzer tool, how to run it and use it catch bugs in Turso. Always load this skill when running this tool4---5 6# Differential Fuzzer7 8Always load [Debugging skill for reference](../debugging/)9 10The differential fuzzer compares Turso results against SQLite for generated SQL statements to find correctness bugs.11 12## Location13 14`testing/differential-oracle/fuzzer/`15 16## Running the Fuzzer17 18### Single Run19 20```bash21# Basic run (100 statements, random seed)22cargo run --bin differential_fuzzer23 24# With specific seed for reproducibility25cargo run --bin differential_fuzzer -- --seed 1234526 27# More statements with verbose output28cargo run --bin differential_fuzzer -- -n 1000 --verbose29 30# Keep database files after run (for debugging)31cargo run --bin differential_fuzzer -- --seed 12345 --keep-files32 33# All options34cargo run --bin differential_fuzzer -- \35  --seed <SEED>           # Deterministic seed36  -n <NUM>                # Number of statements (default: 100)37  -t <NUM>                # Number of tables (default: 2)38  -c <NUM>                # Columns per table (default: 5)39  --verbose               # Print each SQL statement40  --keep-files            # Persist .db files to disk41```42 43### Continuous Fuzzing (Loop Mode)44 45```bash46# Run forever with random seeds47cargo run --bin differential_fuzzer -- loop48 49# Run 50 iterations50cargo run --bin differential_fuzzer -- loop 5051```52 53### Docker Runner (CI/Production)54 55```bash56# Build and run from repo root57docker build -f testing/differential-oracle/fuzzer/docker-runner/Dockerfile -t fuzzer .58docker run -e GITHUB_TOKEN=xxx -e SLACK_WEBHOOK_URL=xxx fuzzer59```60 61Environment variables for docker-runner:62- `TIME_LIMIT_MINUTES` - Total runtime (default: 1440 = 24h)63- `PER_RUN_TIMEOUT_SECONDS` - Per-run timeout (default: 1200 = 20min)64- `NUM_STATEMENTS` - Statements per run (default: 1000)65- `LOG_TO_STDOUT` - Print fuzzer output (default: false)66- `GITHUB_TOKEN` - For auto-filing issues67- `SLACK_WEBHOOK_URL` - For notifications68 69## Output Files70 71All output goes to `simulator-output/` directory:72 73| File | Description |74|------|-------------|75| `test.sql` | All executed SQL statements. Failed statements prefixed with `-- FAILED:`, errors with `-- ERROR:` |76| `schema.json` | Database schema at end of run (or at failure) |77| `test.db` | Turso database file (only with `--keep-files`) |78| `test-sqlite.db` | SQLite database file (only with `--keep-files`) |79 80## Reproducing Errors81 82Always follow these steps83 841. **Find the seed and profile** in the error output:85   ```86   INFO: Starting differential_fuzzer with config: SimConfig { seed: 12345, ..., weight_profile: Writes }87   ```88 892. **Re-run with that seed and profile** (a seed only replays under the same profile):90   ```bash91   cargo run --bin differential_fuzzer -- --seed 12345 --profile writes --verbose --keep-files92   ```93 943. **Read the minimized reproduction first.** On an oracle failure the fuzzer95   writes these files to `simulator-output/`:96   - `minimized.sql` - a shrunken state script plus the shrunken failing97     statement, produced automatically. Start here.98   - `turso-state.sql` / `sqlite-state.sql` - each engine's full state as a99     replayable script, when you need more than the minimized version kept.100   - `test.sql` - every executed statement (the failing one is marked101     `-- FAILED:`). The minimizer falls back to replaying this history when102     the failure depends on how the state was built, not just its contents.103   - `schema.json` - table structure at failure time.104 1054. **Probe the reproduction with `differential_probe`.** It runs a106   statement-per-line script on Turso and SQLite side by side, prints both107   outcomes for every statement, marks divergences, and compares the final108   table contents. Exit code 1 means something diverged.109   ```bash110   cargo run -q -p differential-fuzzer --bin differential_probe -- \111       simulator-output/minimized.sql112   ```113   Use it instead of piping SQL into the two shells: the tursodb shell cannot114   `ATTACH ':memory:' AS aux`, so fuzzer reproductions with an `aux` schema115   only run correctly through the probe. Reading from stdin also works:116   `echo "SELECT ~X'96';" | cargo run -q -p differential-fuzzer --bin differential_probe`.117 1185. **Bisect by editing the script.** Copy `minimized.sql`, simplify one thing119   at a time (replace an expression with a constant, drop a column, drop a120   state line), and re-run the probe after each edit. The divergence marker121   tells you immediately whether the edit kept the bug. This loop usually122   ends at a one-line kernel you can hand to `EXPLAIN` on both engines.123 1246. **Create a regression test** in `.sqltest` (preferred) or `.rs` from the125   kernel. Always load the [Debugging skill for reference](../debugging/).126 127## Understanding Failures128 129### Oracle Failure Types130 1311. **Row set mismatch** - Turso returned different rows than SQLite1322. **Turso errored but SQLite succeeded** - Turso rejected valid SQL1333. **SQLite errored but Turso succeeded** - Turso accepted invalid SQL1344. **Schema mismatch** - Tables/columns differ after DDL135 136### Warning (non-fatal)137 138- **Unordered LIMIT mismatch** - LIMIT without ORDER BY may return different valid rows139 140## Key Source Files141 142| File | Purpose |143|------|---------|144| `main.rs` | CLI parsing, entry point |145| `runner.rs` | Main simulation loop, executes statements on both DBs |146| `oracle.rs` | Compares Turso vs SQLite results |147| `schema.rs` | Introspects schema from both databases |148| `memory/` | In-memory IO for deterministic simulation |149 150## Tracing151 152Set `RUST_LOG` for more detailed output:153 154```bash155RUST_LOG=debug cargo run --bin differential_fuzzer -- --seed 12345156```157