Why This Matters

Running large language models is expensive. Most of the cost comes from GPU time and memory. If you don’t optimize, you burn money fast.

For example, serving a 70B parameter model on A100 GPUs can cost $2,000–$3,000 per day at moderate traffic. That’s $60K–$90K per month for one model. If you run multiple models or regions, the bill explodes.

The good news: you can cut this by 40–60% without hurting quality. Here’s how.

Where the Cost Comes From

  • GPU hours

  • Memory bandwidth

  • Network egress

  • Idle time when GPUs wait for requests

Four Ways to Reduce Cost

1. Batch Requests

Combine multiple requests into one forward pass.

  • Why it works: GPUs are good at parallel work. Batching keeps them busy.

  • How: Use a batcher in your serving stack (vLLM, Triton, or custom).

  • Watch out:

    • At low traffic, batching adds latency.

    • Use dynamic batching with a max wait time (e.g., 10–20 ms).

Impact: Batching can improve throughput by 2–4x. That means fewer GPUs for the same load.

2. Use KV Cache

Store key-value pairs from previous tokens.

  • Why it works: For long prompts, you don’t need to recompute everything.

  • How: Enable KV cache in your model server (vLLM does this well).

  • Watch out:

    • KV cache eats memory. Plan for it.

    • Use eviction policies for long sessions.

Impact: KV cache can cut compute per token by 50–70% for long contexts.

3. Quantize Models

Convert weights to lower precision (INT4, FP8).

  • Why it works: Smaller weights = less memory and faster compute.

  • How: Use AWQ, GPTQ, or TensorRT-LLM for quantization.

  • Watch out:

    • Quality can drop if you push too far.

    • Test on your eval set before going live.

Impact: INT4 can cut memory by 75% and speed up inference by 1.5–2x.

4. Schedule Smart

Use request schedulers and warm pools.

  • Why it works: Avoid cold starts and idle GPUs.

  • How:

    • Keep a small warm pool of GPUs ready.

    • Scale up/down based on traffic.

    • Use bin-packing to fill GPUs.

  • Watch out:

    • Over-scaling kills savings.

    • Under-scaling hurts latency.

Impact: Smart scheduling can save 10–20% on top of batching and quantization.

Trade-offs to Know

  • Batching vs Latency: Great for throughput, but adds wait time.

  • Quantization vs Quality: INT4 saves cost but can hurt accuracy.

  • KV Cache vs Memory: Faster inference, but eats GPU memory.

  • Autoscaling vs Stability: Aggressive scaling can cause cold starts.

Actionable Steps

  • Enable dynamic batching in your serving stack.

  • Add KV cache for long prompts.

  • Test INT4 or FP8 quantization on your eval set.

  • Use autoscaling with warm pools.

  • Track GPU utilization and p95 latency.

References

  • vLLM GitHub

  • Triton Inference Server

  • From my experience: batching + quantization gave 50% cost drop in one project.

Subscribe for weekly AI infra deep dives.