The lede
Cutting our inference bill by 68% by being obsessive about optimization
Cloudflare · made the Sunday
Cloudflare's AI code reviewer drifted from $1.19 a review in April to $2.46 by late August without a single architectural change, and this is the full accounting of getting it to $0.77 — plus a median review time of 1m 32s, down from 4m 09s. The diagnosis is the one every agent operator should expect to find: 77% of the spend was cache writes and reads, context the model had already seen, because each of 16.6 LLM calls per review resent the whole conversation. The fixes are unglamorous and specific — split the system prompt so the shared 11–21k tokens get their own cache breakpoint and every repository can reuse it, move reference files into the prompt instead of spending three turns reading them, delete tools sub-reviewers aren't allowed to call (they tried anyway in up to 76% of sessions, then burned another call recovering), stop the harness from re-reading the conversation to write a summary nobody saw, and notice that GitLab's executor never tells a cancelled job to stop, so 358 of 446 cancelled reviews ran to completion. Only 40% of the saving came from code; the rest came from model and effort choices they A/B'd rather than assumed, including finding Opus 5.5 at medium indistinguishable from high on findings. Read it as the missing manual for the question nobody's spend dashboard answers.