> ## Documentation Index
> Fetch the complete documentation index at: https://docs.10et.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Eval Scripts

> Write eval scripts that produce real gradient for agent improvement

Eval scripts are the reward function for your agents. They measure a metric before and after an agent makes changes. The delta determines whether the change is kept or reverted.

## Anatomy of an Eval Script

An eval script is a bash or TypeScript file that:

1. Runs tests, measurements, or analysis
2. Outputs a JSON object with a primary metric
3. Exits with code 0 (even if the metric is bad)

```bash eval/test-coverage.sh theme={null}
#!/usr/bin/env bash
set -euo pipefail

# Use AGENT_WORKTREE when running in agent context,
# fall back to main repo for manual runs
CLI_DIR="${AGENT_WORKTREE:-$HOME/CascadeProjects/my-project}"

# Run jest with coverage
cd "$CLI_DIR"
npx jest --coverage --coverageReporters=json-summary --silent

# Parse coverage report
COVERAGE_FILE="coverage/coverage-summary.json"
LINE_PCT=$(python3 -c "import json; print(json.load(open('$COVERAGE_FILE'))['total']['lines']['pct'])")

# Output JSON with primary metric
SCORE=$(python3 -c "print(round($LINE_PCT / 100.0, 4))")
echo "{\"coverage_percent\": $SCORE, \"line_pct\": $LINE_PCT}"
```

## The AGENT\_WORKTREE Pattern

<Warning>
  **Critical:** Agents work in isolated git worktrees at `/tmp/tenet-agent-*`. Your eval script must test the agent's changes, not the main branch.

  Always use `AGENT_WORKTREE` when referencing the target repo:

  ```bash theme={null}
  CLI_DIR="${AGENT_WORKTREE:-$HOME/CascadeProjects/my-project}"
  ```

  This uses the worktree when running in agent context, and falls back to the main repo for manual testing.
</Warning>

## Good vs Bad Eval Scripts

### ❌ Bad: Metric at ceiling

```bash theme={null}
# Tests pass rate — already 100%, no gradient
PASSED=$(npx jest --json | jq '.numPassedTests')
TOTAL=$(npx jest --json | jq '.numTotalTests')
echo "{\"pass_rate\": $(echo "$PASSED / $TOTAL" | bc -l)}"
# Always outputs 1.0 — agent can't improve this
```

### ✅ Good: Metric with real gradient

```bash theme={null}
# Coverage percentage — 12.8% baseline, 87% room to improve
npx jest --coverage --coverageReporters=json-summary
LINE_PCT=$(jq '.total.lines.pct' coverage/coverage-summary.json)
echo "{\"coverage_percent\": $(echo "$LINE_PCT / 100" | bc -l)}"
# Outputs 0.128 — every test file moves this number
```

### ❌ Bad: Binary pass/fail

```bash theme={null}
# Does the build succeed? Yes/no
if npm run build; then echo '{"build": 1.0}'; else echo '{"build": 0.0}'; fi
# No gradient — agent gets 1.0 or 0.0, nothing in between
```

### ✅ Good: Continuous signal

```bash theme={null}
# Console.log count — 4461 instances, every cleanup moves the number
COUNT=$(grep -r "console\.\(log\|error\|warn\)" src/ | wc -l)
SCORE=$(python3 -c "print(round(1.0 - min($COUNT, 5000) / 5000, 4))")
echo "{\"quality_score\": $SCORE, \"console_count\": $COUNT}"
```

## Eval Script Requirements

1. **Output JSON** — Must output a single JSON object on stdout
2. **Primary metric** — The JSON must include the metric field matching your agent config
3. **Exit 0** — Even if the metric is bad. Non-zero exit = eval failure, not bad score
4. **Deterministic** — Same code should produce the same score
5. **Fast** — Under 30 seconds is ideal. Maximum is the agent's `time_budget_seconds`
6. **Stderr for logs** — Write debug output to stderr, not stdout

```bash theme={null}
echo "Running coverage..." >&2  # Goes to stderr (logs)
echo '{"coverage_percent": 0.128}'  # Goes to stdout (result)
```

## Composite Metrics

For multi-dimensional quality, create a weighted composite:

```bash eval/code-quality.sh theme={null}
#!/usr/bin/env bash
CLI_DIR="${AGENT_WORKTREE:-$HOME/CascadeProjects/my-project}"

# Multiple signals
TODOS=$(grep -r "TODO\|FIXME" "$CLI_DIR/src" | wc -l)
CONSOLE_LOGS=$(grep -r "console\.log" "$CLI_DIR/src" | wc -l)
ANY_USAGE=$(grep -r ": any" "$CLI_DIR/src" | wc -l)

# Individual scores (0-1)
TODO_SCORE=$(python3 -c "print(round(1.0 - min($TODOS, 50) / 100, 4))")
CONSOLE_SCORE=$(python3 -c "print(round(1.0 - min($CONSOLE_LOGS, 5000) / 5000, 4))")
ANY_SCORE=$(python3 -c "print(round(1.0 - min($ANY_USAGE, 1000) / 1000, 4))")

# Weighted composite
QUALITY=$(python3 -c "print(round($TODO_SCORE * 0.3 + $CONSOLE_SCORE * 0.4 + $ANY_SCORE * 0.3, 4))")

echo "{\"quality_score\": $QUALITY, \"todos\": $TODOS, \"console_logs\": $CONSOLE_LOGS, \"any_usage\": $ANY_USAGE}"
```

## Testing Your Eval

Run it manually first:

```bash theme={null}
# From your project root
bash eval/test-coverage.sh
# {"coverage_percent": 0.128, "line_pct": 12.8}

# With agent worktree simulation
AGENT_WORKTREE=/tmp/test-worktree bash eval/test-coverage.sh
```

Make sure the output is valid JSON and the primary metric has room to improve.