I went looking for dropped requests during a rolling restart and found something more annoying than dropped requests: a deploy that looks perfect and isn't.
Setup is deliberately boring. Two Node/Express replicas behind nginx, ten clients hammering an endpoint that takes three seconds, docker stop on one replica halfway through. The app is the version most of us have shipped at some point, with no signal handling at all:
No SIGTERM handler. Docker sends the signal, Node exits, and anything mid-flight dies with it. I expected a pile of 502s.
The requests did die. nginx caught the upstream connection dropping before any response headers had gone out, so it quietly opened a connection to the other replica and ran the whole thing again. The client never knew.
That behaviour is proxynextupstream, it's on by default, and I want to be fair to it because it is doing exactly what you'd want a reverse proxy to do when a backend disappears mid-request. It is also the reason your dashboard can report a flawless deploy while the thing being deployed is quietly broken, which is a strange position for a metric to be in.
Same zero-error result, same load, same everything. The affected requests took twice as long, because they were executed twice. If you are watching error rate you see nothing. If you are watching p99 you see a spike at every deploy that you have probably learned to ignore.
Three seconds of extra latency is survivable. Doing the work twice might not be, and that depends entirely on what the work is: a retried search query costs you nothing, a retried outbound email costs you a duplicate, and a retried payment authorisation costs you a phone call from someone in finance. nginx has no idea which of those it just re-ran.
Plenty of setups don't have that retry. A Kubernetes Service is iptables or IPVS, and it does not re-run your request. An L4 load balancer won't. A client talking straight to your app certainly won't. Once headers are on the wire, even nginx can't.
There is the pile of 502s I went looking for. Nothing about the app changed. The only difference is whether something upstream was covering for it.
The app-level fix is the one everybody writes about. Stop accepting new connections, let the in-flight ones finish, then exit:
Better. Not fixed. That last failure is stubborn, it showed up on all three runs, and it is the interesting one.
The in-flight requests are safe now. What's left is the requests arriving in the gap between server.close() and the load balancer working out that this instance is gone, and during that gap nginx is still holding the address in its upstream list, so it does the reasonable thing and opens a fresh connection to a socket that has just stopped accepting them, which produces a 502 for a client who did nothing wrong.
No amount of application code fixes that. The app has already done the right thing. The load balancer is the one still pointing at it.
Remove the instance from the load balancer, give the change a second to settle, and only then send SIGTERM:
That's the whole ordering. Stop routing to it, then stop it. In Kubernetes this is what a preStop hook buys you, and it is why preStop: sleep 5 looks like a hack and isn't. The sleep isn't for your app, which is already finished. It's to let the endpoint removal propagate before the container goes away.
My first attempt at the drain test came back with 3, 1 and 1 failures, and I nearly wrote a paragraph explaining that draining doesn't help as much as you'd hope.
It was my bug. I had mounted nginx.conf read-only, so the command that swapped in the drained config failed without complaining, no drain ever happened, and what I had actually done was run the same test twice and then write an explanation for the difference between two identical things. With a writable mount it's zero out of seventy, three runs in a row.
Worth saying out loud because the failure mode is so ordinary: my test harness was broken in a way that produced plausible numbers.