cost

LLM pricing and costs are the economic factors that determine the total expense of running AI applications — including API costs per token, self-hosting infrastructure expenses, and optimization strategies, critical for building sustainable AI products and making build-vs-buy decisions.

What Are LLM Costs?

  • Definition: Total expense of using LLMs in production.
  • Components: API fees, infrastructure, optimization, engineering.
  • Unit: Typically cost per million tokens (input and output separately).
  • Variation: 100× difference between cheapest and most expensive options.

Why Pricing Matters

  • Product Economics: AI features must be profitable.
  • Build vs. Buy: Self-hosting vs. API decision.
  • Architecture Choices: Model routing, caching, batching decisions.
  • Scale Planning: Costs compound at scale.
  • Competitive Position: Lower costs enable lower prices or higher margins.

API Pricing Comparison (2024)

Provider/Model          | Input/1M tk | Output/1M tk | Notes
------------------------|-------------|--------------|---------------
GPT-4o                  | $2.50       | $10.00       | Most capable
GPT-4o-mini             | $0.15       | $0.60        | Cost-optimized
GPT-3.5-turbo           | $0.50       | $1.50        | Legacy
Claude 3.5 Sonnet       | $3.00       | $15.00       | Strong reasoning
Claude 3 Haiku          | $0.25       | $1.25        | Fast, cheap
Gemini 1.5 Pro          | $1.25       | $5.00        | Long context
Gemini 1.5 Flash        | $0.075      | $0.30        | Fastest
Llama 3.1 70B (hosted)  | $0.20-0.80  | $0.20-0.80   | Varies by host
Mistral Large           | $2.00       | $6.00        | European option

Self-Hosting Economics

Infrastructure Costs:

Hardware Option    | Monthly Cost | Models Served
-------------------|--------------|--------------------
RTX 4090 (24GB)    | ~$500 amort. | 7-13B models
A100 40GB          | $2-3K cloud  | Up to 30B
A100 80GB          | $3-4K cloud  | Up to 70B
H100 80GB          | $4-6K cloud  | 70B+ fast inference
8× H100 cluster    | $30-40K      | Any model, high throughput

Break-Even Analysis:

API cost example: $5/M tokens × 10M tokens/day = $50/day = $1,500/month
H100 cost: ~$5,000/month
Break-even: ~100M tokens/day for H100
Below this: API often cheaper
Above this: Self-host saves money

Cost Optimization Strategies

Caching:

Common queries → Cache responses
Hit rate of 20% → 20% cost reduction
Semantic caching: Similar queries hit cache
Implement: Redis, custom cache layer

Model Routing:

Simple queries → Cheap/small model (90% of traffic)
Complex queries → Expensive/large model (10% of traffic)
Potential savings: 60-80%

Prompt Optimization:

Before: 2,000 token system prompt
After: 500 token optimized prompt
Savings: 75% on input tokens

Techniques:
- Compression
- Remove redundancy
- Batch instructions

Output Control:

max_tokens: Set appropriate limits
Stop sequences: End early when possible
JSON mode: Structured output (often shorter)

Batching:

Real-time: Process individually (higher per-request cost)
Batch: Accumulate, process together (lower per-request cost)
When acceptable latency allows, batch for savings

Cost Tracking

What to Measure:
- Tokens per request (input + output).
- Requests per user/feature.
- Cost per user action.
- Cost per successful outcome.

Implementation:

class CostTracker:
    def __init__(self):
        self.costs = defaultdict(float)
    
    def record(self, user_id, feature, 
               input_tokens, output_tokens, model):
        cost = calculate_cost(
            input_tokens, output_tokens, model
        )
        self.costs[user_id] += cost
        self.costs[feature] += cost
        self.log(user_id, feature, cost)

Cost by Use Case

Use Case              | Typical Cost      | Optimization
----------------------|-------------------|-------------------
Chat (1 turn)         | $0.001-0.01       | Cache, small model
Code completion       | $0.0001-0.001     | Small model, prefix caching
Document summary      | $0.01-0.10        | Batch, smaller model
RAG (search + answer) | $0.005-0.05       | Cache embeddings
Agent (multi-step)    | $0.10-1.00        | Limit retries, cheaper tools

Cost Control Architecture

LLM Cost Optimization — Dollars per Million Tokens model choice × quantization × caching × routing × batching = 10-100× cost reduction at equal quality API Pricing ($/1M output tokens, July 2026) Haiku $1.25 GPT-4o-mini $0.60 Sonnet 3.5 $15 GPT-4o $10 Opus $75 o1-pro $150 Self-hosted (vLLM on H100) Llama-3 70B: ~$0.40/MTok at full utilization Llama-3 8B quantized: ~$0.05/MTok Output tokens cost 3-5× more than input tokens (decode is memory-bandwidth bound) Cost Reduction Levers 1. Model routing (5-10×) cheap model for easy queries 2. Prompt caching (2-5×) reuse system prompt KV cache 3. Batch API (2×) async processing at 50% discount 4. Prompt compression (1.5-3×) fewer input tokens (LLMLingua) 5. Self-host + quantize (10-50×) open model, INT4, full GPU util 6. Distillation train small model on large's outputs Cost Math Example Scenario: 1M requests/day, 500 in + 200 out tokens avg GPT-4o: $2.5K input + $2K output = $4.5K/day GPT-4o-mini: $75 input + $120 output = $195/day Self-hosted 8B: ~$35/day (1× H100 at $1.50/hr) Saving: $4.5K → $195 (route) → $35 (self-host) = 130× cost reduction, quality depends on task Optimization Strategy by Stage Day 1: Prototype use best model, ignore cost establish quality baseline Week 2: Evaluate test smaller models on eval set find cheapest at same quality Month 1: Route classify queries, route to tiers 80% queries → mini model Month 3: Self-host fine-tune open model, deploy maximize tokens/$/quality The frontier model is the eval baseline, not the production model — always ask: can a cheaper model do this task? LLM cost optimization is the new cloud cost optimization — the companies that solve it first win on unit economics.

LLM pricing and costs are the foundation of AI product economics — understanding and optimizing costs determines whether AI features are sustainable at scale, making cost engineering as important as prompt engineering for production AI systems.

Go deeper with CFSGPT

Get AI-powered deep-dives, save terms, and run advanced simulations — free account.

Create Free Account