Token Budget — How to Allocate Your 200K Window

200K sounds like a lot. Two copies of To Live (the Chinese novel runs ~120K characters) would fit. But in production LLM apps, 200K is never enough — five context layers fight over the same pool, every extra retrieval chunk costs a turn of history, every extra tool costs a block of memory.

What 200K actually buys you

JR omni-report's daily-jobs routine — by turn 5 of the conversation:

# tested: 2026-04-26 · model: claude-sonnet-4-6 · max_context: 200K

Layer 1  System instruction              ~800 tok      (cached)
Layer 2  Tool definitions (6 tools)     ~1,100 tok     (cached)
Layer 3  History (5 轮对话 + scratchpad)  ~8,500 tok
Layer 4  Retrieved context              ~32,000 tok    (LinkedIn 30 jobs 全文)
Layer 5  User input                       ~120 tok
─────────────────────────────────────────────────────
                                       ~42,520 tok    (使用 21%)

Output budget (max_tokens)              ~4,096 tok
─────────────────────────────────────────────────────
Effective budget                       ~46,616 tok    (23%)

Five turns in and 23% is gone. Run 15 more turns, history quadruples, and at 30K retrieval per turn you cap out at turn 20. 200K isn't "infinite" — it's a turn budget.

Not every token costs the same — pricing dictates the strategy

Anthropic's current pricing puts tokens in four tiers (relative ratios using Claude Sonnet 4.6 as baseline):

If system + tools are 2000 tokens and never change, cache hits make them feel like 200 tokens of budget. Anthropic's docs say it plainly: "prompt caching can reduce costs by up to 90%." Output costs 5× input, so max_tokens=4096 is a direct cost cut, not just brevity.

Five budget techniques (ROI-sorted)

1. Cache the fixed layers (highest ROI)

Add cache_control: {type: "ephemeral"} to system + tools, 5-minute TTL:

# tested: 2026-04-26 · anthropic@0.40.0
system=[
    {
        "type": "text",
        "text": SYSTEM_PROMPT,  # ~800 tok
        "cache_control": {"type": "ephemeral"}
    }
],
tools=[
    *[{**tool, "cache_control": {"type": "ephemeral"}}
      for tool in TOOL_DEFS]  # ~1100 tok
]

Cost: first call pays +25% cache-write tax; next 5 minutes those layers cost 10% per call. A high-frequency workload (≥6 calls/min) saves 60-90% daily.

Catch: cache key is byte-exact. One character changes in system/tools and cache shatters.

2. Summarize history (next ROI)

History blows first. Pattern:

• Keep most recent N turns verbatim (5-10)
• Push older turns through a cheap model (Haiku) to compress, stuff summary at end of system prompt marked [历史摘要]

LangChain's ConversationSummaryBufferMemory wraps this. Mem0 / Letta are heavier stacks (chapter 6).

Catch: summaries lose info. Tasks needing exact numbers / proper nouns must keep originals or re-fetch via RAG.

3. Lazy-load tools

15 tools × 200 tokens = 3000 tokens on every call. But 80% of tasks only touch 2-3.

• Phase 1 ship a meta tool, searchTools(query), returning relevant tool names
• Phase 2 when called, inject the schema then
• Claude Code defaults to this — ToolSearch for on-demand loading

Catch: needs multi-turn flow. One API call can't do it.

4. Rerank retrieval (preview of chapter 5)

Retrieval is the biggest token sink. bi-encoder pulls 50 → cross-encoder reranks to 10 → LLM-as-judge picks 3. Same 90% recall, tokens drop from 25K to 1.5K.

5. Output budget cap

max_tokens=4096 vs 8192 — neither runs full, but the model's generation strategy shifts based on max_tokens. Give it 8K and it leans long-form; 4K and it self-compresses.

JR experience: JSON-output tasks at max_tokens=2048 produce valid JSON more reliably than 8192 — with a long budget the model gets chatty and slips prose into JSON.

Real JR case: omni-report routine budget strategy

The 17 omni-report routines run on Claude Code Remote, with a hidden constraint: stream idle timeout ≈ 10 minutes — no output for that long and you get cut off. Every routine has to chop big tasks into phases with forced commits. JR AI Visibility Weekly:

Phase 0: 准备 + 读上游           (~30 sec, 5K tok)
Phase 1: 写骨架 + commit          (~1 min, 2K tok)
Phase 2: 4 batch × 5 query        (~6 min × 4, 30K tok)
  └─ 每 batch 完成 → Edit → commit + push
Phase 3: 仪表盘 + 洞察 + 行动清单  (~2 min, 10K tok)
Phase 4: Notion 全文同步           (~30 sec, 8K tok)

Each phase outputs <8K, together they're 60K+. One LLM call doing the whole thing trips idle timeout — splitting into phases is both budget and reliability strategy.

Aggressive vs Generous — Trade-off

JR's rule: first app starts generous; only after it works + has eval set + ran 100 real queries does it go aggressive. Going aggressive on day one drops context, and eval can't tell you why.

Takeaway

200K context isn't a budget. It's a turn capacity that 5 layers fight over. Token cost isn't uniform — cache hit is 1×, output is 50×, cache write is 12.5×. The engineering core for production apps: send the first 2 layers (system + tools) through cache, the 3rd (history) through summaries, the 4th (retrieval) through rerank.

---

References

1. Anthropic. Pricing — input / output / cache 4 档计费.
2. Anthropic. Prompt caching documentation — 5 分钟 TTL ephemeral cache + 90% 成本节省说明.
3. Anthropic. Long context tips — 长 context 信息组织建议.
4. Anthropic. Models overview — context window 容量 + 模型对比.
5. LangChain. ConversationSummaryBufferMemory — 摘要 history 实现.

Production case: JR Academy omni-report routines — 「骨架 + 渐进填充 + 每段就 commit」的 phase 拆分模式，应对 stream idle timeout 的 budget 策略.