A practical look at what happens to your margins when every feature calls a model

Someone in a meeting says "we should just use AI for that." Everyone nods. It sounds free — the model already exists, the API is one HTTP call away, and the demo you built over the weekend worked flawlessly. Three months later you're staring at a bill that grows in lockstep with the users you fought so hard to acquire, wondering why success suddenly feels like a liability.

This is the part of AI engineering nobody demos. The magic is real, but so is the meter running behind it. Understanding that meter — before you scale, not after — is the difference between an AI feature that funds your company and one that quietly bleeds it.

Tokens are the unit of everything

Every interaction with a language model is billed in tokens. A token is roughly three-quarters of a word — so a hundred words costs you somewhere around 130 tokens. You pay for what you send (input) and what you get back (output), and crucially, output almost always costs several times more than input.

That asymmetry matters more than people expect. A feature that sends a short prompt and receives a long response has a very different cost profile from one that sends a huge document and gets back a single word. When you're estimating what something will cost at scale, you can't just count "requests" — you have to count tokens, and you have to count input and output separately.

Here's the pattern that catches teams off guard: the cost is invisible during development. You test with ten requests. Ten requests cost a fraction of a cent. Everything feels free. Then you launch, ten requests become ten million, and the fraction of a cent becomes a number that shows up in board meetings.

Not all models cost the same — by a lot

The single biggest lever on your AI costs is which model you reach for, and the spread between the cheapest and most capable models is enormous — often more than an order of magnitude per token.

The instinct is to always use the best model. It gives the best answers, so why compromise? Because most tasks don't need the best model. Classifying a support ticket, extracting a date from a sentence, summarizing a paragraph, checking whether text is on-topic — these are handled perfectly well by a fast, cheap model. Reserving your most expensive model for the genuinely hard reasoning, and routing everything else to a cheaper one, can cut your bill by ninety percent without users noticing any drop in quality.

This is the first discipline: match the model to the difficulty of the task, not to your desire to use the shiniest thing available. A tiered approach — cheap model by default, expensive model only when the task demands it — is how you keep quality high and cost sane at the same time.

The techniques that actually move the needle

Beyond model choice, a handful of practical techniques reliably lower cost:

Prompt caching. Most applications send the same instructions on every request — the same system prompt, the same context, the same examples. Caching lets you pay full price for that shared prefix once and a steep discount on every subsequent call. If your prompts have a large, stable portion, this alone can be a dramatic saving.

Trim your context. Models charge for every token you send, including the ones that don't help. Dumping an entire document into a prompt when the model only needs two relevant paragraphs is paying to send noise. Retrieving and sending only the relevant slice cuts input costs directly.

Cap your output. Output is the expensive half. Setting a sensible maximum on how much the model can generate prevents runaway responses and keeps costs predictable. If you need ten questions, ask for ten — don't leave the door open for fifty.

Batch where latency allows. If you're processing things that don't need an instant answer — generating content overnight, enriching a database — batching many items together is often cheaper than firing off individual real-time requests.

Building the cost into the product, not around it

The teams that get this right don't treat cost as an afterthought to optimize once the bill hurts. They design for it from the start, and it shapes the product itself.

Sometimes that means a generous free tier backed by a cheap model, with the expensive model gated behind a paid plan. Sometimes it means giving users a fixed budget of AI actions per month rather than unlimited access. Sometimes it means being honest that a feature is too expensive to offer for free and pricing it accordingly. These aren't compromises on the vision — they're the difference between a vision that survives contact with real usage and one that collapses under it.

The mental model that helps: know your cost per action, and know what you charge for it. If a single AI-powered action costs you a couple of cents and you're giving it away to hundreds of thousands of users, that's not a growth strategy, it's a countdown. If that same action costs a couple of cents and sits inside a plan someone pays real money for, you have a business.

The math you should do before you build

Before committing to an AI feature, run a rough calculation that most teams skip:

Estimate the tokens a typical request consumes, input and output separately. Multiply by the per-token price of the model you plan to use. That gives you the cost of a single action. Now multiply by the number of times you expect it to happen per month at the scale you're aiming for. Compare that number to the revenue those users generate.

If the AI cost is a small fraction of the revenue, you have room to breathe. If it's a large fraction, you have a design problem to solve now — cheaper model, caching, tighter context, a paywall — while it's still a spreadsheet and not a crisis. Ten minutes of arithmetic before you build can save you a very uncomfortable conversation later.

"Just use AI" is a beginning, not an answer

There's nothing wrong with reaching for AI to solve a problem — it's genuinely the right tool for an expanding range of jobs. The mistake is treating "use AI" as the end of the engineering conversation rather than the start of it.

The real work begins after that decision: which model, cached how, fed what context, capped at what length, priced into which plan. That's not bureaucracy slowing down the magic. That's the engineering that turns an impressive demo into a feature you can actually afford to ship to everyone who wants it.

The model is powerful, cheap per call, and available to anyone. Which is exactly why knowing how to wield it economically — not just technically — is becoming one of the more valuable skills an engineer can have.