Something interesting happened when engineers at Radar gave their AI diagnostic agent a set of runbooks and architecture documents before turning it loose on broken Kubernetes clusters. Accuracy jumped from 60% to 86%.
That is a meaningful gap. Two diagnoses out of three correct becomes more than four out of five. When each missed diagnosis means an engineer spending hours tracking down a root cause manually, the difference adds up fast.
The experiment is worth understanding. Kubernetes is the system most modern cloud applications run on. It manages containers (self-contained packages of code and their dependencies) and keeps them running across groups of servers at scale. When something in Kubernetes breaks, diagnosing the problem correctly requires specialized knowledge. Site reliability engineers, the professionals whose job is keeping software systems healthy and available, can spend hours on a single incident.
Radar ran 50 simulated fault scenarios on a live Amazon EKS cluster, which is Amazon’s managed Kubernetes service. In one condition, the AI agent had its standard tools and nothing else. In the other, the agent also had context, specifically runbooks with documented procedures for known problem types, architecture notes, and background about the specific environment. The research used Claude Code alongside Microsoft’s SREGym benchmark to measure results. The agents with context reached a correct diagnosis within two minutes on 86% of faults. The agents without context managed 60%.
Experienced engineers will recognize exactly why this happened. An agent diagnosing an unfamiliar system has to reconstruct what “normal” looks like from scratch on every attempt. An agent that has the runbook for a problem type, knows the expected behavior of each service, and has seen the architecture before is working from a much stronger starting position.
The practical takeaway is not about which AI model to choose. It is about what you feed it. Teams that have invested in clear runbooks and documented architecture will get significantly more out of AI-assisted operations tools than teams that have not. Your documentation is no longer just for onboarding new engineers. It has become input for the automated tools working alongside them.
Good docs make better AI. That might be the most underrated argument for writing things down.
Want to explore how AI-assisted DevOps could benefit your engineering team? Let’s talk.