A GPU can pass every cloud provider health check and still corrupt a training job. Netflix confirmed this at scale. They built a four-layer reliability system to address it and published their full engineering approach this week.
The stakes at Netflix are not abstract. Their recommendation engine, the system that decides which titles appear on your home screen, trains on GPUs. Those runs can span weeks across hundreds of machines operating in lockstep, each synchronizing its computation with every other at every step. The job advances only as fast as its slowest participant. One degraded GPU in the cluster, and the whole operation slows.
Why Standard Health Checks Fall Short
Cloud provider health checks confirm that a machine is reachable. That’s all. They do not verify that each GPU, its memory, and its high-speed network interconnect are healthy enough to handle a sustained workload. A GPU can respond to a diagnostic query and still fail once a job allocates memory and starts running real computation. A network adapter can appear fully present while operating at a fraction of its expected bandwidth, slowing every machine in the cluster without triggering a single alert.
Even the diagnostic tools themselves can fail silently, hanging when the driver hangs or quietly stopping when a dependency goes missing after a routine driver update. A check that stops running without reporting anything is worse than no check, because it gives false confidence while problems accumulate.
The Four-Layer Solution
Netflix built their reliability model around four layers. Each catches what the previous one misses. The first layer runs device-level checks before and during jobs, scanning each GPU for memory errors, thermal problems, and degraded network links. The second layer sits in the scheduler, validating each machine before it receives work and draining machines that show warning signs. The third layer lives inside the training loop, using checkpointing and elastic workers so a job can survive losing a machine mid-run without restarting from scratch. The fourth layer is an observability system covering metrics, service-level objectives, and failure attribution that confirms whether the first three are doing their jobs.
The underlying principle is to assume the hardware can lie. Netflix engineered every layer to fail loudly when its own inspection machinery breaks, so there is no such thing as a silently failing check.
This matters beyond Netflix’s scale. Any company running AI workloads on cloud GPU instances faces a version of this problem. Most smaller teams rely on provider defaults and discover the gap only when a long job fails well into the run, losing hours or days of compute time.
The four-layer model Netflix describes is not their exclusive invention. It is how production GPU infrastructure should be managed, at any scale, and the pattern applies whether you are running a short experiment or a multi-week training job.
Want to explore how better infrastructure reliability practices could benefit your AI operations? Let’s talk.