Datadog studied more than 35,000 production AI agent sessions. In the median session, the agent was actually doing something, calling a language model or executing a tool, for 20% of its runtime. The other 80% was idle time. It was just waiting.
Most teams building AI agents don’t know this about their own workloads. That gap shows up on the infrastructure bill.
The instinct when standing up an AI agent is to give it a dedicated server or an always-on compute instance. Keep it warm, keep it ready. That instinct comes straight from traditional web application development, where idle compute is an acceptable trade-off for fast response times. AI agents work differently. They do expensive bursts of work (calling large language models, running tools, waiting on external APIs) and then go quiet for stretches in between. Running always-on infrastructure for an agent that sits idle four out of every five minutes is a straightforward mismatch.
Datadog’s research makes a practical case: start with serverless architecture, and let production data tell you if you need something else.
Serverless, in plain terms, means the platform provisions compute only when your agent needs to do something, and scales back to zero when it does not. You pay for active execution time, not idle minutes. For workloads that are 80% idle, that difference is significant.
A 2026 Gartner survey found that 83% of senior IT leaders say they need infrastructure upgrades to run production-grade AI agents at scale. Many are defaulting to provisioning more dedicated capacity upfront. Starting there often means paying for idle time from day one.
The better starting approach involves three steps. First, build an execution profile before committing to an architecture. Record how often your agent runs, how long each active burst lasts, and what the longest individual step takes. That profile tells you whether serverless fits before you spend money on alternatives. Second, use durable serverless workflows for longer-running agents. Durable workflows, a pattern where the system saves its progress between steps and releases compute during waits, let an agent pause while waiting for a human to approve something or for an external API to respond, then resume without holding allocated resources the whole time. Third, move specific steps to dedicated compute only when measured data shows that startup latency or sustained utilization actually justifies it.
The trap most teams fall into is overprovisioning early out of concern for performance, then discovering months later that their agents barely use what they have allocated.
Serverless is not right for every agent workload. Some agents need persistent connections. Some have latency requirements tight enough that cold start time, the small delay when serverless spins up fresh compute, creates a real problem. The argument isn’t that serverless always wins. The argument is that it’s the right default to test against, and you should need evidence to move away from it.
Building AI agents on the right infrastructure from the start reduces costs and makes scaling straightforward when adoption grows across your organization.
Want to explore how serverless AI infrastructure could benefit your business? Let’s talk.