intent driven development

Keeping Tests Current

Tests that don’t match current behavior are worse than no tests — they provide false confidence. This page shows how chant detects and fixes test drift.

The Drift Problem

At Acme, 23% of tests are out of sync with current behavior:

Drift Type	Count	Risk
Tests asserting removed behavior	45	High — false positives
Tests missing new parameters	32	Medium — incomplete coverage
Tests using deprecated mocks	28	Low — maintenance burden
Tests with outdated docstrings	89	Low — documentation confusion

Documentation Specs for Test Mapping

Documentation specs track the relationship between tests and behavior:

File: .chant/specs/2026-01-22-001-doc.md

---
type: doc
status: ready
labels:
  - payments
  - test-mapping
watches:
  - src/payments/refund.py
  - tests/payments/test_refund_flow.py
target_files:
  - docs/testing/refund-test-mapping.md
---

# Document refund test-to-behavior mapping

## Purpose

Track which tests cover which behaviors in the refund module.
This mapping enables drift detection when code changes.

## Acceptance Criteria

- [ ] Each public method mapped to its tests
- [ ] Each error code mapped to its test
- [ ] Coverage gaps identified
- [ ] Document written to target file

Generated Mapping Document

File: docs/testing/refund-test-mapping.md

# Refund Module Test Mapping

Generated by spec 2026-01-22-001-doc

## Method: `RefundService.process_refund()`

| Behavior | Test | Status |
|----------|------|--------|
| Full refund succeeds | `test_full_refund_on_completed_transaction` | ✓ |
| Partial refund calculates balance | `test_partial_refund_correct_remaining_balance` | ✓ |
| Returns to original payment method | `test_refund_to_original_payment_method` | ✓ |
| Triggers confirmation email | `test_refund_confirmation_email_triggered` | ✓ |

## Method: `RefundService.validate_refund_request()`

| Behavior | Test | Status |
|----------|------|--------|
| Rejects amount > original | `test_refund_exceeds_original_returns_400` | ✓ |
| Rejects 90+ day old transactions | `test_refund_older_than_90_days_rejected` | ✓ |
| Blocks disputed transactions | `test_refund_on_disputed_transaction_blocked` | ✓ |

## Error Codes

| Code | Description | Test |
|------|-------------|------|
| REFUND_EXCEEDS_ORIGINAL | Amount > transaction | ✓ Covered |
| TRANSACTION_TOO_OLD | > 90 days | ✓ Covered |
| DISPUTED_TRANSACTION | Active dispute | ✓ Covered |
| INSUFFICIENT_BALANCE | Already refunded | ✗ Missing |

## Coverage Gaps

- `INSUFFICIENT_BALANCE` error code has no dedicated test

Drift Detection

The chant drift command compares current code against documentation specs:

chant drift 2026-01-22-001-doc

Drift Detection Report — 2026-01-22-001-doc

Watched files changed since last check:
  src/payments/refund.py: 3 changes detected

Drift Analysis:

1. NEW PARAMETER: `reason` added to process_refund()
   - Added in commit abc123 (2026-01-25)
   - No tests verify reason parameter
   - Recommendation: Add test for reason validation

2. BEHAVIOR CHANGE: Authorization thresholds updated
   - $100 → $150 for auto-approval
   - $1000 → $1500 for manager approval
   - Tests still assert old thresholds
   - Recommendation: Update authorization tests

3. NEW ERROR CODE: REFUND_ALREADY_PROCESSED
   - Added for idempotency check
   - No test coverage
   - Recommendation: Add idempotency test

Summary: 3 drift issues detected
  - 1 missing parameter test
  - 1 outdated assertion
  - 1 missing error test

Automated Drift Checks

Marcus sets up a weekly drift report:

File: .github/workflows/drift-check.yml

name: Weekly Drift Check

on:
  schedule:
    - cron: '0 9 * * 1'  # Monday 9am

jobs:
  drift:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Check all doc specs for drift
        run: |
          chant drift > drift-report.txt

      - name: Notify on drift
        if: ${{ steps.drift.outputs.drift_count > 0 }}
        run: |
          echo "Test drift detected - see drift-report.json for details"

Fixing Drift with Update Specs

When drift is detected, create a spec to fix it:

chant add "Fix test drift in refund module"

File: .chant/specs/2026-01-27-001-fix.md

---
type: code
status: ready
labels:
  - payments
  - tdd
  - drift-fix
depends_on:
  - 2026-01-22-001-doc
target_files:
  - tests/payments/test_refund_flow.py
  - tests/payments/test_refund_edge_cases.py
---

# Fix test drift in refund module

## Problem

Drift detection found 3 issues (spec 2026-01-22-001-doc):
1. Missing test for `reason` parameter
2. Outdated authorization thresholds
3. Missing test for REFUND_ALREADY_PROCESSED error

## Acceptance Criteria

- [ ] Add test for reason parameter validation
- [ ] Update authorization threshold tests ($100→$150, $1000→$1500)
- [ ] Add test for REFUND_ALREADY_PROCESSED error
- [ ] Update test-to-behavior mapping document
- [ ] All tests passing
- [ ] Drift check passes with no issues

Drift Detection Output

After fixing drift:

chant drift 2026-01-22-001-doc

Drift Detection Report — 2026-01-22-001-doc

Watched files changed since last check:
  src/payments/refund.py: No new changes

Drift Analysis:
  No drift detected ✓

All tests aligned with current behavior.

Continuous Drift Prevention

To prevent drift from accumulating, Marcus adds a drift check to PR validation:

# .github/workflows/pr-check.yml
- name: Check for drift
  run: |
    # Check drift on all doc specs
    chant drift

This fails the PR if code changes would cause test drift without updating tests.

Before/After: Drift Metrics

After implementing drift detection:

Metric	Before	After	Target
Test drift rate	23%	4%	<5% ✓
Tests asserting removed behavior	45	3	—
Missing parameter tests	32	5	—
Outdated docstrings	89	12	—

End-to-End Recap

The full TDD transformation journey:

Week 1: Identify testing problems        → 3 teams, 3 different approaches
Week 2: Align chant with TDD workflow    → Specs = test plans
Week 3: Write test specs, execute        → 27 new tests, coverage 45% → 86%
Week 4: Enforce standards, detect drift  → Consistent quality, <5% drift

All test artifacts — specs, coverage reports, drift analyses — are tracked in git. Any engineer can trace from a test back to its spec, and from a spec to the business requirement that drove it.

Chant Documentation