Why your p99 isn't your p99

Your p99 is a lie, and this note is about why. We spent four weeks chasing a latency spike that wasn't in Prometheus, wasn't in our traces, and wasn't in the customer's logs. It was in the tokenizer.

## The setup

Seven-node serving cluster, vLLM, reasonable ingress. Customer reported ~3s p99 on a workload we'd clocked at 900ms. Our dashboards said 900ms. Theirs said 3s. Both were right.

## The discovery

Token-level latency — the time between the first token and each subsequent one — was never in our metrics. We measured request-level latency and called it done. The customer's client was timing out on gaps, not totals.

## The fix

Emit a histogram per generation, bucketed by inter-token gap. Alert on p95 of that, not p99 of request time. Six lines of code, one unhappy quarter.