Open-source benchmark · MIT · runs locally on Ollama
Measure how many tokens your agent wastes re-reading its own history.
The default agent loop re-sends the whole conversation every turn — cumulative input tokens grow O(n²). Cap the window to save money and recall rots to 0% by turn 10. context-clock measures both, on your own workload, in under 60 seconds — and proves the fix: 21× fewer input tokens with retrieved memory, identical across six models.
# clone & run on your own hardware — no API keys $ git clone github.com/bolnet/context-clock $ python -m context_clock.run --until-rotted --turns 100 # → watch input tokens go O(n²) and recall rot to 0%
O(n²)
input-token growth with full-context replay
0%
recall by turn 10 once you cap the window
21×
fewer input tokens with retrieved memory
6 / 6
models confirm the identical curve
What it measures
Two ways to run out of road — and context-clock charts both.
There's no free lunch with raw context. Let it grow and you pay quadratically; cap it and the model forgets. context-clock reproduces each failure mode deterministically, then measures the way out.
Grow it → quadratic cost
Re-read the whole transcript every turn and input tokens pile up as n²/2. By turn 100 a single gpt-5.4 session has processed 1.33M input tokens — for one conversation.
Cap the window to bound cost and the oldest turns truncate. Recall decays in a staircase — 100% to 0% by turn 10 — identical from a 3B model to a 671B one. Truncation is mechanical.
100% → 0%
recall, by turn 10 · capped @1024 · every model
The proof · linear vs quadratic
A parabola becomes a straight line.
Same workload, run two ways, both holding 100% recall: re-send everything (raw) vs retrieve only what's needed (memory). Cumulative input tokens — the part memory actually controls.
t100
turn
1.33M
raw input tokens
61.8K
memory input tokens
21.5×
fewer input tokens
raw — re-send all (quadratic → 1.33M)memory — retrieve top-1 (linear → 61.8K)gpt-5.4 · native window · 100% recall both sides
Input × is set by workload geometry, not the model — every model traces the same curve: 5.6× (t24) → 11× (t50) → 16× (t75) → ~21–22.5× (t100). Cost reduction varies with provider caching and answer verbosity; tokens are the universal number.
Run it on your workload
No API keys. No cost. 60 seconds.
context-clock runs locally on Ollama — fork it, swap in your own memo shapes and models, and watch the exact token bloat and recall rot on your own hardware. Every number on this page is reproducible.
# 1 · the rot stress test — context grows until recall dies $ python -m context_clock.run --until-rotted --turns 100
# 3 · any model via OpenRouter — same harness $ python -m context_clock.run --provider openrouter --model openai/gpt-5.4
99
tests — reproduce every number here
~200
flat tokens / call with memory, t1 → t100
MIT
open source · no lock-in · fork freely
6
models benchmarked · open + closed
Under the hood · 100% reproducible
The benchmark, in detail.
A deterministic, single-fact needle-in-a-haystack workload — the arc reproduces on every model. Here's what sits under the number, and you can re-run all of it locally.
0models benchmarked · open + closed
0turns per session
0tests · every number reproducible
0recall held, all turns
Per-call context — what the model reads each turn
raw · turn 100
8,709
memory · any turn
~209
Raw climbs every turn toward the window; memory stays flat at ~209 tokens from turn 1 to turn 100.
The gap widens with session length
t24
5.6×
t50
11.0×
t75
16.2×
t100
21.5×
Input-token reduction (gpt-5.4). A straight line vs a parabola → the ratio is unbounded.
Method · honest scope
One unique code per memo (needle-in-a-haystack), retrieved top-1, no distractors — isolates "does the system still have the fact?"
Native context window — models run at full capacity; capping is shown only to reproduce the cap-and-rot failure mode.
100% recall = the right memo is reliably found and read — not a claim about hard multi-hop or conflicting-fact retrieval.
temperature 0, deterministic; real billed cost from OpenRouter usage.cost, never list-price math.
You've measured the waste. Now stop paying for it.
context-clock proves the problem; Attestor is the open-source memory layer that fixes it — flat ~200 tokens per call, 21× fewer input tokens, 100% recall, two API calls.