The Cold Start Nobody Warned You About

Scale-to-zero sounds great on paper. Pay nothing when idle, scale up on demand. Every cloud pitch deck has this slide. What the deck doesn’t show is the user who submitted a request, waited eight seconds, got nothing, and left.

That’s a cold start. And in ML inference, it’s not a footnote — it’s a product decision you probably made by accident.

What’s Actually Happening

When a model isn’t actively serving traffic, your infrastructure shuts it down. GPU deallocated. Container stopped. Next request comes in, and the system has to spin everything back up: allocate hardware, load the container, load the model weights into GPU memory, warm up the serving framework, and then — finally — process the request.

For a small text classifier, that might be two or three seconds. For a 7B parameter model, you’re looking at 15-30 seconds. For anything larger, longer.

Users don’t read loading spinners as “infrastructure provisioning.” They read them as broken.

The Part People Get Wrong

Most teams treat cold starts as a latency problem. It’s not. It’s an availability problem dressed up as latency.

A slow response and a cold-start response feel identical to the user, but they’re different failure modes with different fixes. Tuning inference speed doesn’t help if the GPU isn’t even warm yet. You need a different lever.

The common ones: minimum replica counts (keep at least one instance always hot), predictive scaling based on traffic patterns, or a lightweight ping-on-schedule job that keeps the endpoint alive during expected usage windows.

None of these are clever. All of them work.

When to Actually Use Scale-to-Zero

Scale-to-zero isn’t wrong — it’s just misapplied. It’s appropriate for:

• Batch jobs and async processing where latency doesn’t matter
• Internal tools with known usage patterns and tolerant users
• Dev and staging environments where cost matters more than speed
• Models that serve rare, non-time-sensitive requests

It is not appropriate for your customer-facing product, your real-time chatbot, or anything where the first second of response time shapes whether the user trusts the system.

The Decision You Owe Your Team

Cold start behavior should be an explicit decision in your serving architecture, not an inherited default from whatever cloud service you deployed on. Write it down. “This endpoint scales to zero because it’s batch-only.” Or: “This endpoint keeps two warm replicas because it’s customer-facing.”

If your team doesn’t know the answer, that’s not an infrastructure gap. That’s a communication gap, and the user is paying for it.

Warm your models like you mean it, or scale to zero on purpose. Just stop doing it by default.