On-prem inference is cheaper than you think.

A 70B model serving one million requests a day costs you more than you think on cloud, and less than you think on your own rack. We ran the numbers. Here they are.

Most TCO comparisons for AI infrastructure are either napkin math or vendor spreadsheets. The napkin math underestimates on-prem operational cost; the vendor spreadsheet overestimates cloud commit discounts. Neither is useful if you're the one signing.

Over the last eighteen months we've run this comparison for seven customers. The patterns are consistent enough to be worth publishing — with the caveat that your numbers will differ.

## The shape of the bill

**On the cloud side**, inference cost is dominated by GPU-hours, not storage or egress. For a 70B model with reasonable context lengths, you're paying for H100-class hardware whether you use it or not — because cold-start on these models is too slow for interactive workloads. That means your effective utilization caps the bill.

**On the on-prem side**, the bill is front-loaded. Hardware amortization dominates year one. Power, cooling, and operations dominate years two through five.

### A quick sanity check

```
$ cat tco-cloud.txt
GPU-hours / year 76,650 (8x H100, 24/7, 90% util)
$ / GPU-hour $2.49
compute $190,857
egress + obs $14,200
--------------------------------
annual cloud bill $205,057

$ cat tco-onprem.txt
capex (amortized 5y) $92,000
power @ $0.11/kWh $21,800
cooling $8,900
ops + support $38,000
--------------------------------
annual on-prem bill $160,700

delta -$44,357 (-22%)
```

## Where the assumptions hide

The honest parts of any TCO comparison are where you admit what you're guessing. Our model assumes 90% utilization — if yours is 40%, the on-prem math stops working. It assumes a 5-year amortization — if your finance team wants 3, the numbers slide. And it ignores the switching cost of rebuilding your team, which is rarely zero.