Tokenizer Changes & Your Cost Baseline

What changed, stated precisely, and why it is not a defect.

The relevant sentence is on the vendor's public rate card as of October 2026: Claude 4.7 and later models, plus Claude Mythos Preview, use a newer tokenizer that "contributes to their improved performance on a wide range of tasks" and "produces approximately 30% more tokens for the same text." Claude Sonnet 4.6 and earlier use the previous tokenizer. The page adds the clause that does most of the operational work: "the exact increase depends on the content and workload shape."

Read that as a disclosed trade, not an error. A finer-grained vocabulary gives the model more, smaller pieces to reason over, and model authors judged the capability gain worth the token count. What it is not is a change you can ignore on the finance side, because a token is not a unit of text — it is a unit of the vendor's segmentation of text, and the mapping between the two just moved by roughly a third. Every quantity you denominated in tokens inherited that move without anybody editing a config. The tokenization concept covers the mechanics; what matters here is that the unit is theirs, not yours.

The arithmetic that turns a price cut into a price rise.

Hold the work constant — the same prompt text, the same answer length in words — and vary only the model. The rate card falls; the token count rises; the product is what you pay.

Same work, same text. USD per million tokens, October 2026.

Model        Tokenizer  Input  Output  Tokens vs 4.6  Relative bill
-----------  ---------  -----  ------  -------------  -------------
Opus 4.6     previous   $5.00  $25.00          1.00×          1.00
Opus 4.7     newer      $5.00  $25.00          1.30×          1.30
Opus 5.5     newer      $4.00  $20.00          1.30×          1.04

Opus 4.6 → 4.7:  price unchanged, bill up    ~30%
Opus 4.6 → 5.5:  price down 20%, bill up     ~4%
Opus 4.7 → 5.5:  price down 20%, bill down   ~20%

Three readings, and teams get burned by different ones. The first row pair is the trap for anyone who upgraded within a generation on the strength of an unchanged rate card: identical pricing, roughly a third more spend, and no line item to point at. The second is the trap for anyone planning a migration from a pre-4.7 model onto the current tier and booking the 20% as a saving — on these published numbers it is approximately a wash before you account for anything else. The third is the honest comparison, because both sides are quoted in the same unit, and it recovers the full headline cut.

Two things are deliberately excluded from that table, and both can dominate it. Capability changes how many steps a task takes — a model that gets there in twelve tool calls instead of twenty is cheaper per task even at a higher rate per token, which is the effect fewer tokens versus cheaper tokens puts in closed form. And reasoning effort changes output volume independently of tokenization. The table is not a forecast; it is the correction you must apply before your forecast means anything.

Four other things denominated in tokens that moved at the same time.

Spend is the consequence people notice. These four are the ones that cause incidents, because each is a limit rather than a bill, and limits fail abruptly.

• Context capacity, measured in text. A fixed token window holds about 23% less prose after a 30% inflation (1 ÷ 1.30). A prompt assembly step that fitted a document set into 180,000 tokens with margin now truncates it, and truncation in an agent is usually silent. Derive the budget from characters or pages and convert, rather than hard-coding a token figure — see context budgeting, and effective versus advertised context for the separate reason the usable window is smaller than the stated one.
• Tokens-per-minute rate limits. A TPM quota is a text-throughput quota in disguise. The same quota carries about 23% less work after the change, so a workload that ran at 80% of its limit now runs at about 104% of it and starts getting 429s under no change in traffic. Rate limits and provider capacity has the backoff side; the point here is that the trigger moved without a deploy.
• Output caps. A max_tokens of 4,096 chosen because it comfortably held your longest report now holds about a quarter less of it, and the failure mode is a truncated answer that looks like a model quality regression. This one is especially nasty in evals, where a truncated response scores as a wrong response and the post-mortem goes looking for the wrong cause.
• Length-priced thresholds. Where a tier changes price by prompt length — Claude Haiku 5.5 charges 5× on requests whose prompt passes 100,000 tokens — a token inflation moves the crossing earlier in the run. The same conversation that used to stay on the cheap side of the line for twenty steps crosses it at around step fifteen. Cost, quality and latency works that cliff through.

Note the common shape: in every case a number that was chosen with margin lost its margin, and nothing in your change log explains it. This is the same class of failure as unpinned vendor defaults — a value you treated as a constant turned out to belong to someone else.

Measure your own ratio; do not use 30%.

The vendor says the increase depends on content and workload shape, and that caveat is load-bearing. Tokenizer changes are not uniform across content types: English prose, source code, structured tool output and non-Latin scripts each have their own ratio, and a corpus that is 60% JSON tool results will not move like a corpus of English documents. If your traffic is multilingual, multilingual agents explains why the per-language spread is often the widest term of all.

The measurement is cheap and you should do it before the migration, not after the invoice.

# Sample real traffic, bucket it by content class, count both ways.
# Any token-counting endpoint or local tokenizer will do; what matters
# is that the SAME text is counted under both tokenizers.

for bucket in [english_prose, source_code, json_tool_results, zh_prose]:
    old = count_tokens(bucket, model="pre-4.7-model")
    new = count_tokens(bucket, model="current-model")
    ratio[bucket] = new / old

# Weight by each bucket's share of YOUR token volume, not by count.
blended = sum(ratio[b] * volume_share[b] for b in buckets)

Use the blended ratio to restate your pre-migration baseline in new-tokenizer units before you compare anything. Keep the per-bucket ratios too: they tell you where a context budget is most at risk of truncating, and they frequently reveal that one content class — usually machine-generated tool output — accounts for most of the inflation and is also the easiest thing to shrink.

Re-baseline the accounting onto the one invariant.

Cost per completed task is the only figure that means the same thing on both sides of a tokenizer boundary, which is the practical reason cost per task is the unit this section keeps returning to. Concretely:

• Make cost-per-successful-task the reported metric and demote cost-per-token to a diagnostic. Unit economics and cost attribution both assume a stable denominator; this is what keeps them honest.
• Annotate the boundary on every token-denominated chart. A vertical marker reading "tokenizer change" on the day the model switched prevents a quarter of arguments about a step change that is a unit change.
• Re-derive token budgets from text budgets. Store the limit as "the last 40 turns plus 12 retrieved passages", compute the token figure at assembly time, and let it move with the tokenizer instead of silently breaking against it.
• Re-check every commitment denominated in tokens. A provisioned-throughput or spend commitment bought in token units purchases fewer tasks after an inflation — see provisioned throughput and commitments, and renegotiate on tasks where the contract allows it.
• Re-price your eval suite. Cost of evaluation scales directly with token volume, and an eval budget set in dollars against the old unit buys roughly 23% fewer runs — which quietly reduces your statistical power at exactly the moment you are trying to measure a migration.
• Leave the caching maths alone. Cache read and write prices are multipliers on the base input rate, so the breakeven in reads is unchanged by tokenization. Prompt caching is one of the few things on this page you do not have to recompute.

The migration checklist.

Fold this into whatever you already run for model deprecation and migration, because a tokenizer change is a migration risk that does not look like one — the API is identical, the code compiles, and the regression is in units.

• Confirm from the vendor's own documentation whether the source and target models share a tokenizer. Do not infer it from version proximity; the boundary sat between 4.6 and 4.7 in this case, which is as small a version step as exists.
• Measure the blended and per-bucket ratios on sampled production traffic.
• Restate the old baseline in new units, then compare. Report the price change and the unit change as two separate numbers, never as one net figure.
• Re-fit every token-denominated limit: context assembly budget, max_tokens, truncation thresholds, TPM headroom, spend alerts, length-priced tier boundaries.
• Re-run the eval suite at the new output cap specifically to catch truncation scored as wrongness.
• Record the ratio and the date in the runbook. The next tokenizer change will arrive the same way this one did — in a footnote.