Checklist before putting AI into a CI pipeline

The checklist

• Feedback loop. The assistant receives the real output of the build and the tests, and each round is logged.
• Independent oracle. Code and tests are not produced from the same source by the same model.
• Frozen tests. The test directory is unchanged by the AI step, proven by a diff in CI.
• Iteration cap. The loop has a maximum number of rounds, and hitting it fails the job.
• Pinned model. A specific model version is requested, and the one used is written into the output metadata.
• Transcript. The full session record is stored as a pipeline artifact.
• No skips. Adding a test-skipping annotation fails the build.
• No prompts. The job cannot ask for permission, and its allowed actions are listed in advance.

These assume the base commit of the change is available in CI as $BASE_SHA and that the checkout has enough history to diff against it.

1. Feedback loop from build and test results

The assistant should work against what the compiler and the test runner print, not against its own account of what the code does. Pass the real output back as the next input and write each round to the log: the input given, the change produced, the command run and its exit code. A loop with no record of its rounds cannot be reviewed afterwards.

2. Independent oracle

Tests generated from the code, or from the same prompt and model as the code, agree with it by construction: a misunderstanding shared by both passes silently. Take expected behaviour from something independent of the implementation: a written specification, an existing reference, or tests written by a different author. If a model writes tests at all, give it only the specification, not the implementation, and have a person review the result against the specification.

3. Freeze the test directory, and check it by diff

Telling the assistant not to touch the tests is a request. A diff in CI is a control. Fail the job if the change touches the frozen paths. Replace tests/ with your directory.

changed=$(git diff --name-only "$BASE_SHA"...HEAD -- tests/)
if [ -n "$changed" ]; then
  echo "frozen test directory modified:" >&2
  echo "$changed" >&2
  exit 1
fi

A deliberate change to a frozen test is a separate change with its own review, not part of the AI step.

4. Cap the iterations

Bound the loop in the script that runs it, so the limit does not depend on the model behaving. --max-budget-usd adds a spending ceiling to a print-mode call.

max=5
for i in $(seq 1 "$max"); do
  ./ci/run-tests.sh && exit 0        # your own build-and-test entry point
  <your claude -p call, with the test output as input>
done
echo "no passing run after $max rounds" >&2
exit 1

5. Pin the model version and record it

A model alias moves to a newer model over time, so behaviour can change with no change in your repository. Request a full model name with --model, take its value from one place, and write it, with the tool version, into the metadata that is stored with the output.

cat > ai-run-meta.json <<EOF
{
  "model_requested": "$AI_MODEL",
  "cli_version": "$(claude --version)",
  "commit": "$(git rev-parse HEAD)",
  "date_utc": "$(date -u +%Y-%m-%dT%H:%M:%SZ)"
}
EOF

With --output-format stream-json the initial system/init event reports the model in use. Compare it with the value requested, after checking the field names in your version’s actual output. Pinning guarantees the model name you ask for; confirm with your provider how long a given name stays available.

6. Keep the transcript as an artifact

The JSON result carries a session_id. Local sessions are stored as ~/.claude/projects/<project>/<session>.jsonl; on a CI runner, check the actual home directory and project directory name before relying on that path. Upload the file as a build artifact, or capture stream-json output to a file. Do not use --no-session-persistence in a job whose transcript you need: with it, nothing is saved. Treat the transcript as source-bearing: it holds prompts, file contents and command output, so apply the same access and retention rules as to the code.

7. Ban skip annotations

The cheapest way to make a failing test pass is to stop it running. Fail the build if a change adds a skip or ignore marker. Adjust the patterns to your languages; this list is a starting point, not a complete one.

if git diff -U0 "$BASE_SHA"...HEAD | grep -E '^\+[^+]' \
   | grep -nE '@Ignore|@Disabled|pytest\.mark\.(skip|xfail)|\.skip\(|\bxit\(|\bxdescribe\(|t\.Skip\(|#\[ignore\]'; then
  echo "test-skipping annotation added" >&2
  exit 1
fi

This complements the frozen directory: it also catches skips added to tests that live next to the code.

8. Unattended means no prompts

A job nobody watches must not wait for permission, and must not be able to widen its own access. Use a mode that denies instead of asking and name what is allowed. See the CLI tiers and permission modes by kind of work.

claude -p "<task>" \
  --model "$AI_MODEL" \
  --permission-mode dontAsk \
  --allowedTools "Read,Edit,Bash(./ci/run-tests.sh)" \
  --disallowedTools "Bash(git push *)" \
  --output-format json

Run the job with credentials limited to what it needs, and without push rights to the protected branch.

Unofficial notes, not affiliated with or endorsed by Anthropic. Command names, flags and behaviour change between versions; the official documentation and the tool's own help output are authoritative.