<div class="block-paragraph_advanced"><p><span style="vertical-align: baseline;">Every support engineer has a case that stays with them. The one that comes up at team lunches. The one where, even after it's resolved, you find yourself staring at a packet capture at midnight just to make sure you </span><span style="font-style: italic; vertical-align: baseline;">really</span><span style="vertical-align: baseline;"> understand what happened. This is that case.</span></p>
<p><span style="vertical-align: baseline;">It started in January 2026 with a support ticket that read deceptively simply: pods are intermittently failing to reach the Kubernetes API server. The error was a familiar-looking timeout. The customer had already ruled out their own operators. What none of us knew yet was that we were looking at three separate bugs — in three different layers of the stack — all conspiring to produce the same symptom at the same time.</span></p>
<p><span style="vertical-align: baseline;">By the time it was over, around 800 developers had been blocked from deploying to production for weeks on end, and GKE clusters across europe-west had ground to a halt. In a race to fix the problem, we involved engineers across Google Kubernetes Engine (GKE), Private Service Connect, networking, hostnet, kernel, and the Linux upstream community, who together tracked down not one but three distinct root causes. And the final fix landed not as a config change or a feature flag, but as a patch to the Linux kernel itself.</span></p>
<p><span style="vertical-align: baseline;">This is the story of how we got there.</span></p>
<h3><strong style="vertical-align: baseline;">Part one: The alarm bell that wouldn't stop</strong></h3>
<p><span style="vertical-align: baseline;">The customer ran a large multi-cluster GKE environment in Google Cloud's europe-west region. Their clusters used Private Service Connect — Google Cloud's mechanism for exposing the GKE control plane privately — and had a rich ecosystem of system-level workloads: Twistlock security scanners, External Secrets Operator for secrets management, Cilium for network policy, as well as several internal Kubernetes operators.</span></p>
<p><span style="vertical-align: baseline;">The first errors appeared in one of its user acceptance testing (UAT) clusters in europe-west, and a few days later in a second cluster in the same region. They looked like this:</span></p>
<p><span style="vertical-align: baseline;">Application logs — pods across multiple namespaces</span></p></div>
<div class="block-code"><dl>
<dt>code_block</dt>
<dd><ListValue: [StructValue([('code', "TimeoutError: Connect call failed ('1.2.3.4', 443)\r\ndial tcp 1.2.3.4:443: i/o timeout\r\n\r\n# 1.2.3.4 is the cluster's internal VIP for the kube-apiserver,\r\n# routed through Private Service Connect"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fcfb12a2950>)])]></dd>
</dl></div>
<div class="block-paragraph_advanced"><p><span style="vertical-align: baseline;">Initially the errors were intermittent — spikes in the logs, not a hard outage. The customer suspected their own code. Their team had recently deployed a custom operator built with the </span><span style="font-style: italic; vertical-align: baseline;">kopf</span><span style="vertical-align: baseline;"> framework, and it was the first thing they looked at. But the logs showed the errors had started eight days before the operator was ever deployed. When they shut it down completely, the timeouts kept coming.</span></p></div>
<div class="block-pull_quote"><div class="uni-pull-quote h-c-page">
<section class="h-c-grid">
<div class="uni-pull-quote__wrapper h-c-grid__col h-c-grid__col--8 h-c-grid__col-m--6 h-c-grid__col-l--6
h-c-grid__col--offset-2 h-c-grid__col-m--offset-3 h-c-grid__col-l--offset-3">
<div class="uni-pull-quote__inner-wrapper h-c-copy h-c-copy">
<q class="uni-pull-quote__text">The issue persists even after completely scaling down the External Secrets Operator. The errors are appearing from customer workloads, and system-level pods we haven't touched.</q>
</div>
</div>
</section>
</div>
</div>
<div class="block-paragraph_advanced"><h4><span style="vertical-align: baseline;">The growing list of suspects</span></h4>
<p><span style="vertical-align: baseline;">When a case like this arrives, the first instinct is to ask: what changed? The customer's timeline gave us several candidates, and we worked through them methodically.</span></p>
<ul>
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
<p role="presentation"><strong style="vertical-align: baseline;">The custom operator (kopf):</strong><span style="vertical-align: baseline;"> Ruled out immediately — errors predated deployment by a week and continued after shutdown.</span></p>
</li>
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
<p role="presentation"><strong style="vertical-align: baseline;">External Secrets Operator:</strong><span style="vertical-align: baseline;"> Removed entirely on customer request. Timeouts continued in customer workloads that relied on Kubernetes control plane (KCP) connectivity, and cluster-internal system pods.</span></p>
</li>
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
<p role="presentation"><strong style="vertical-align: baseline;">Security agents:</strong><span style="vertical-align: baseline;"> Uninstalled. Same result. We were now watching </span><span style="font-style: italic; vertical-align: baseline;">system-level Kubernetes components</span><span style="vertical-align: baseline;"> fail to reach their own control plane.</span></p>
</li>
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
<p role="presentation"><strong style="vertical-align: baseline;">conntrack table exhaustion:</strong><span style="vertical-align: baseline;"> One theory was that too many half-open connections were filling the OS connection tracking table on the control plane nodes. We checked. The conntrack table was at roughly 50% capacity — of a default maximum of 1.3M entries. Not even close to exhaustion.</span></p>
</li>
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
<p role="presentation"><strong style="vertical-align: baseline;">Control plane overload:</strong><span style="vertical-align: baseline;"> We scaled up the KCP VM size. kube-apiserver latencies looked healthy. No improvement on the timeout side.</span></p>
</li>
</ul>
<p><span style="vertical-align: baseline;">What made this case genuinely strange was the scope. We confirmed the timeouts were happening across five different clusters, affecting DaemonSets, Deployments, and system pods alike. External clients — developer laptops, GitHub Actions runners — were also hitting the same wall. This was not a broken workload. This was something wrong with the path between </span><span style="font-style: italic; vertical-align: baseline;">everything</span><span style="vertical-align: baseline;"> and the API server.</span></p>
<h4><span style="vertical-align: baseline;">The timeline so far</span></h4>
<ul>
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
<p role="presentation"><strong style="vertical-align: baseline;">Day 1:</strong><span style="vertical-align: baseline;"> First I/O timeout errors appear (One of the UAT clusters in europe-west3. Intermittent, easy to dismiss.)</span></p>
</li>
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
<p role="presentation"><strong style="vertical-align: baseline;">Day 10:</strong><span style="vertical-align: baseline;"> Errors become persistent (Frequency increases significantly. The issue is no longer dismissible.)</span></p>
</li>
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
<p role="presentation"><strong style="vertical-align: baseline;">Day 15:</strong><span style="vertical-align: baseline;"> Case escalated — P0 bug filed (Multiple clusters confirmed affected. ~800 developers blocked. Production releases are frozen. Formal escalation begins.)</span></p>
</li>
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
<p role="presentation"><strong style="vertical-align: baseline;">Day 20:</strong><span style="vertical-align: baseline;"> All obvious suspects ruled out (Custom operator, security agents, conntrack exhaustion, control plane resizing — all cleared.)</span></p>
</li>
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
<p role="presentation"><strong style="vertical-align: baseline;">Day 22:</strong><span style="vertical-align: baseline;"> Packet captures collected (Captures taken simultaneously at the control-plane node NIC and from inside the kube-api container. The investigation enters its most critical phase.)</span></p>
</li>
</ul>
<p><span style="vertical-align: baseline;">With the obvious suspects cleared, and the scope undeniably wide, we did what you do when the application layer has nothing left to say: We went to the packets.</span></p>
<h3><span style="vertical-align: baseline;">Part two: Tracing the ghost in the wire</span></h3>
<p><span style="vertical-align: baseline;">To understand what the packet captures revealed, you first need to understand the architecture that a GKE pod traverses when it makes an API call. In a Private Service Connect-based GKE cluster, the path is deceptively long for something that looks like a local connection.</span></p></div>
<div class="block-image_full_width">
<div class="article-module h-c-page">
<div class="h-c-grid">
<figure class="article-image--large
h-c-grid__col
h-c-grid__col--6 h-c-grid__col--offset-3
"
>
<img
src="https://storage.googleapis.com/gweb-cloudblog-publish/images/updated_architecture_official_icon.max-1000x1000.png"
alt="image1">
</a>
</figure>
</div>
</div>
</div>
<div class="block-paragraph_advanced"><p><span style="vertical-align: baseline;">The key component here is Private Service Connect. When a pod dials </span><span style="vertical-align: baseline;">1.2.3.4:443</span><span style="vertical-align: baseline;">, it is not actually going directly to the control plane; that IP is a virtual endpoint. The connection is intercepted by Google's networking fabric, NAT'd through a Private Service Connect endpoint, passed through an internal passthrough Network Load Balancer, and ultimately delivered to the KCP VM, where Envoy acts as a local proxy in front of the actual kube-apiserver process.</span></p>
<p><span style="vertical-align: baseline;">Private Service Connect adds a proxy protocol header to each new connection, so the kube-apiserver can see the real client IP. </span><span style="font-style: italic; vertical-align: baseline;">This will turn out to be important later.</span></p>
<h4><span style="vertical-align: baseline;">What the packets revealed</span></h4>
<p><span style="vertical-align: baseline;">When we captured simultaneously at the KCP VM's NIC (the PIGv interface, Google's programmable virtual NIC) and from inside the kube-apiserver container, we saw something striking.</span></p>
<p><span style="vertical-align: baseline;">SYN packets from the pods </span><span style="font-style: italic; vertical-align: baseline;">were</span><span style="vertical-align: baseline;"> arriving at the KCP VM's NIC. They were getting that far. But instead of receiving a SYN-ACK in response — the expected second step of a TCP handshake — the client was getting back an out-of-place ACK. Not an RST. Not silence. An ACK that logically belonged to a different, earlier connection.</span></p>
<p><span style="vertical-align: baseline;">Wireshark, the network protocol analyzer, was flagging these streams with a label that initially seemed innocuous: "TCP Port numbers reused."</span></p>
<p><strong style="vertical-align: baseline;">Wireshark — packet stream showing the failed handshake</strong></p></div>
<div class="block-code"><dl>
<dt>code_block</dt>
<dd><ListValue: [StructValue([('code', 'Time Source Port → Destination Port Info\r\n09:13:41 10.0.13.77 53426 → 10.117.0.2 443 [SYN] Seq=0 Win=42600\r\n09:13:41 10.117.0.2 443 → 10.0.13.77 53426 [SYN, ACK] Seq=0 Ack=1 Win=64768\r\n ... (data exchange, connection completes)\r\n09:13:42 10.117.0.2 443 → 10.0.13.77 53426 [TCP Retransmission]\r\n09:13:42 10.117.0.2 443 → 10.0.13.77 53426 [TCP Retransmission]\r\n ... (many retransmissions of the same ACK)\r\n\r\n# 2 minutes later, same port 53426 is reused for a new connection\r\n09:13:44 10.0.13.77 53426 → 10.117.0.2 443 [SYN] Seq=1498 ← NEW connection\r\n (expected: SYN-ACK)\r\n (received: ACK Seq=1498 Ack=1025 — belongs to PREVIOUS stream)\r\n09:13:44 10.0.13.77 53426 → 10.117.0.2 443 [SYN] Retransmit...\r\n09:13:44 10.0.13.77 53426 → 10.117.0.2 443 [SYN] Retransmit...\r\n → eventually: i/o timeout'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fcfb12a27d0>)])]></dd>
</dl></div>
<div class="block-paragraph_advanced"><p><span style="vertical-align: baseline;">The pattern was clear. Something was causing the KCP VM to respond to a fresh SYN with an ACK that belonged to a </span><span style="font-style: italic; vertical-align: baseline;">different connection that had ended minutes ago</span><span style="vertical-align: baseline;">. The old connection had used the same source port (53426 in the example above), and the KCP VM had never fully let go of it.</span></p>
<p><span style="font-style: italic; vertical-align: baseline;">"The KCP server sends FIN,ACK; client responds with FIN,ACK and then ACK. But KCP receives ACK before FIN,ACK. KCP server never ACKs the FIN,ACK — continuous retransmission of ACK. RST does not clear state."</span></p>
<h4><strong style="vertical-align: baseline;">The "Invalid ACK" spike</strong></h4>
<p><span style="vertical-align: baseline;">While one thread of the investigation was analyzing packet captures, a.k.a., pcaps, another team was digging into internal metrics for the Private Service Connect infrastructure — specifically the user space packet switch (USPS), Google's software switching layer that handles NAT and proxy protocol header injection for Private Service Connect connections.</span></p>
<p><span style="vertical-align: baseline;">They found something alarming in the metrics for Andromeda, Google's networking fabric. A counter called "Proxy protocol drop: invalid ACK packet" had been flat for months — and then, in February 2026, it spiked sharply upward to several drops per second across the affected europe-west regions.</span></p>
<p><span style="vertical-align: baseline;">That date matched a USPS version rollout. This version included a change which was a fix for a </span><span style="font-style: italic; vertical-align: baseline;">different</span><span style="vertical-align: baseline;"> bug but something about it had changed the behavior of how Private Service Connect handled a specific class of packet.</span></p>
<h4><strong style="vertical-align: baseline;">Challenge ACKs: The RFC 5961 mechanism</strong></h4>
<p><span style="vertical-align: baseline;">To understand the next piece of the puzzle, we need to talk about a TCP security mechanism called the challenge ACK, defined in RFC 5961.</span></p>
<p><span style="vertical-align: baseline;">Imagine a TCP connection has closed — gone through the FIN/ACK dance and terminated. The server-side (here, the KCP VM) may still briefly remember the connection's sequence numbers in its TCP state machine. Now imagine that shortly afterward, a new client connection arrives on the same source port and destination port (the same 5-tuple). From the kernel's perspective, this </span><span style="font-style: italic; vertical-align: baseline;">could</span><span style="vertical-align: baseline;"> be a stale packet from the old connection, or it could be an attacker trying to inject data into what the kernel thinks is still an open session.</span></p>
<p><span style="vertical-align: baseline;">RFC 5961 says: when you receive a SYN that matches an existing or recently-closed connection's remembered state, don't accept it blindly. Send a challenge ACK — an acknowledgment that carries the </span><span style="font-style: italic; vertical-align: baseline;">last known sequence number</span><span style="vertical-align: baseline;"> of the old connection. If the new client is legitimate, it will see this unexpected ACK, recognize the connection is stale, and send a RST to clear the state. The server then accepts the next SYN normally.</span></p>
<p><strong style="vertical-align: baseline;">The challenge ACK flow — how it's supposed to work</strong></p></div>
<div class="block-code"><dl>
<dt>code_block</dt>
<dd><ListValue: [StructValue([('code', '# Step 1: New client sends SYN on a reused port\r\nClient → Server: SYN (seq=X, same 5-tuple as an old connection)\r\n\r\n# Step 2: Server is in FIN_WAIT_2, sends challenge ACK\r\nServer → Client: ACK (ack=old_seq — "is this really you?")\r\n\r\n# Step 3: Client recognizes the stale state, resets it\r\nClient → Server: RST\r\n\r\n# Step 4: State is cleared. Server accepts the next SYN normally.\r\nClient → Server: SYN (new connection)\r\nServer → Client: SYN-ACK ← handshake succeeds'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fcfb12a3f90>)])]></dd>
</dl></div>
<div class="block-paragraph_advanced"><p><span style="vertical-align: baseline;">This mechanism works perfectly — </span><span style="font-style: italic; vertical-align: baseline;">when the challenge ACK is allowed through</span><span style="vertical-align: baseline;">. But that was exactly what the new USPS change was blocking.</span></p>
<h4><span style="vertical-align: baseline;">The culprit: A well-intentioned fix</span></h4>
<p><span style="vertical-align: baseline;">The change in </span><span style="vertical-align: baseline;">USPS</span><span style="vertical-align: baseline;"> had been designed to fix a legitimate security issue in Private Service Connect's proxy protocol handling. Part of its effect was to drop packets it classified as "invalid ACKs" during the connection handshake phase — specifically, ACKs that arrived before the initial SYN-ACK had been sent, which in normal circumstances would be anomalous.</span></p>
<p><span style="vertical-align: baseline;">Unfortunately, challenge ACKs look exactly like this: They arrive from the server before the new handshake has completed, in response to a SYN. From the USPS's new perspective, they looked like invalid packets. They were dropped silently.</span></p>
<p><span style="vertical-align: baseline;">Why this broke GKE specifically: Private Service Connect uses NAT to share a single public endpoint across many client pods. This means many different pods — running on different nodes, from different namespaces — can all end up using the same source port after NAT, creating a high rate of 5-tuple collisions. The challenge ACK mechanism was being triggered far more often than it would be in a normal direct-connection scenario. The USPS fix hadn't </span><span style="font-style: italic; vertical-align: baseline;">introduced</span><span style="vertical-align: baseline;"> challenge ACKs — they had always been there. It had introduced the </span><span style="font-style: italic; vertical-align: baseline;">dropping</span><span style="vertical-align: baseline;"> of them, which broke a flow that GKE's TCP stack had silently depended on to clear stale state.</span></p>
<h4><span style="vertical-align: baseline;">Mitigation: Rolling back the change</span></h4>
<p><span style="vertical-align: baseline;">In February, we rolled back the </span><span style="vertical-align: baseline;">USPS change</span><span style="vertical-align: baseline;">, restoring the old behavior where challenge ACKs were passed through instead of dropped. The rate of "Proxy protocol drop: invalid ACK packet" dropped back toward baseline almost immediately.</span></p>
<p><span style="vertical-align: baseline;">By early March , 2026, the customer confirmed that all affected clusters had recovered. No more timeout errors.</span></p>
<p><span style="vertical-align: baseline;">But the rollback wasn't a fix. It was a band-aid. It restored the behavior the system had </span><span style="font-style: italic; vertical-align: baseline;">accidentally depended on</span><span style="vertical-align: baseline;">. The deeper question — why was the KCP VM in FIN_WAIT_2 for over two minutes in the first place? — was still completely unanswered.</span></p>
<p><span style="vertical-align: baseline;">That question led to Part 3. And it's where things got genuinely strange.</span></p>
<h3><strong style="vertical-align: baseline;">Part three: Three bugs, one crime scene</strong></h3>
<p><span style="vertical-align: baseline;">The mitigation bought us time, but it left a question hanging in the air: Why did the KCP VM keep connections in </span><span style="vertical-align: baseline;">FIN_WAIT_2</span><span style="vertical-align: baseline;"> for so long? A well-behaved TCP stack processes a FIN, acknowledges it, and closes the connection. The KCP VM was failing to do this, leaving half-open connections that persisted for more than two minutes — long enough for the port to be reused by Private Service Connect's NAT layer, long enough to cause the challenge ACK scenario we'd uncovered.</span></p>
<p><span style="vertical-align: baseline;">To answer this, we needed to understand not just one bug, but three distinct failure modes that had intersected at the worst possible moment.</span></p>
<h4><span style="vertical-align: baseline;">Bug #1: Packet reordering at the PIGv</span></h4>
<p><span style="vertical-align: baseline;">The first piece was discovered through careful analysis of simultaneous captures at two points: the KCP VM's host NIC (inside the physical host running the VM) versus the PIGv — the virtual NIC as seen by the VM.</span></p>
<p><span style="vertical-align: baseline;">At the host NIC level, packets arrived in perfect order. But by the time they crossed through USPS and appeared at the PIGv, they had been subtly reordered.</span></p>
<p><span style="vertical-align: baseline;">This happens because of how Private Service Connect's </span><span style="font-style: italic; vertical-align: baseline;">miss-thread processing</span><span style="vertical-align: baseline;"> works. When Private Service Connect intercepts a new flow it hasn't seen before (a "miss"), it processes that packet through a slower path that adds the proxy protocol header and sets up NAT state. Subsequent packets in the same flow take a fast path. The consequence: a FIN or RST packet belonging to a flow in mid-close might arrive slightly out of order relative to the ACKs that followed it.</span></p>
<p><strong style="vertical-align: baseline;">What the KCP VM actually received (reordered sequence)</strong></p></div>
<div class="block-code"><dl>
<dt>code_block</dt>
<dd><ListValue: [StructValue([('code', "# What should have happened:\r\nClient → Server: FIN, ACK (client initiates close)\r\nServer → Client: ACK (server acknowledges client FIN)\r\nServer → Client: FIN, ACK (server initiates its own close)\r\nClient → Server: ACK (client acknowledges server FIN)\r\n → Connection fully closed. Both sides free.\r\n\r\n# What the KCP VM actually received (reordered by PIGv/USPS):\r\nClient → Server: ACK (arrives first — out of order)\r\nClient → Server: FIN, ACK (arrives second, but kernel sees ACK first)\r\nServer → Client: FIN, ACK (server sends close, enters FIN_WAIT_1)\r\nClient → Server: FIN, ACK (old, now-reordered FIN from client)\r\n → Server should ACK this. But it doesn't..."), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fcfb12a3f50>)])]></dd>
</dl></div>
<div class="block-paragraph_advanced"><p><span style="vertical-align: baseline;">The reordering put the client's FIN after the server had already sent its own FIN. The server entered FIN_WAIT_1, then FIN_WAIT_2 after seeing the client's ACK of the server's FIN. But then the client's </span><span style="font-style: italic; vertical-align: baseline;">own</span><span style="vertical-align: baseline;"> FIN arrived — and the server's kernel ignored it.</span></p>
<h4><span style="vertical-align: baseline;">Bug #2: Envoy's half-close and the frozen timer</span></h4>
<p><span style="vertical-align: baseline;">Why couldn't the operating system clean up on its own? There is a Linux kernel parameter called </span><span style="vertical-align: baseline;">tcp_fin_timeout</span><span style="vertical-align: baseline;"> that governs exactly this: how long a socket in </span><span style="vertical-align: baseline;">FIN_WAIT_2</span><span style="vertical-align: baseline;"> waits before being forcibly closed. On the KCP VM, this was set to 60 seconds.</span></p>
<p><span style="vertical-align: baseline;">But </span><span style="vertical-align: baseline;">tcp_fin_timeout</span><span style="vertical-align: baseline;"> only applies to </span><span style="font-style: italic; vertical-align: baseline;">orphaned</span><span style="vertical-align: baseline;"> sockets — sockets that no process is currently holding open. Envoy, the proxy that sits in front of kube-apiserver on the KCP VM, was performing a half-close: it called </span><span style="vertical-align: baseline;">shutdown(SHUT_WR)</span><span style="vertical-align: baseline;">, which closes the write side of the connection but keeps the socket open so it can still receive data. This is perfectly valid TCP behavior and is how graceful half-duplex termination is supposed to work.</span></p>
<p><span style="vertical-align: baseline;">The side effect: As long as Envoy held the socket with </span><span style="vertical-align: baseline;">SHUT_WR</span><span style="vertical-align: baseline;">, the kernel did not consider it orphaned. </span><span style="vertical-align: baseline;">tcp_fin_timeout</span><span style="vertical-align: baseline;"> never started. The socket sat in </span><span style="vertical-align: baseline;">FIN_WAIT_2</span><span style="vertical-align: baseline;"> indefinitely, waiting for a client FIN it would never be able to process correctly.</span></p>
<p><strong style="vertical-align: baseline;">Kernel socket state — captured via netstat just before a new connection</strong></p></div>
<div class="block-code"><dl>
<dt>code_block</dt>
<dd><ListValue: [StructValue([('code', 'Proto Local Address Foreign Address State\r\ntcp 10.128.0.5:443 10.0.13.77:53426 FIN_WAIT_2\r\n# This socket has been in FIN_WAIT_2 for > 120 seconds.\r\n# tcp_fin_timeout=60 has not fired because the socket is not orphaned.\r\n# Envoy holds it open with SHUT_WR.\r\n\r\n# 2 minutes later, Private Service Connect NAT reuses port 53426 for a new pod connection.\r\n# The kernel sees the new SYN on this 5-tuple and sends a challenge ACK.\r\n# USPS drops the challenge ACK. → i/o timeout.'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fcfb12a0910>)])]></dd>
</dl></div>
<div class="block-paragraph_advanced"><h4><span style="vertical-align: baseline;">Bug #3: The Linux kernel's silent failure</span></h4>
<p><span style="vertical-align: baseline;">But there was still a missing piece. Even with the reordered FIN, a healthy kernel should still process a received FIN in </span><span style="vertical-align: baseline;">FIN_WAIT_2</span><span style="vertical-align: baseline;"> and move to </span><span style="vertical-align: baseline;">TIME_WAIT</span><span style="vertical-align: baseline;">. The socket should eventually close. Why wasn't it?</span></p>
<p><span style="vertical-align: baseline;">This led us deep into the Linux TCP state-machine implementation. When the reordered client FIN arrived at the KCP VM, the kernel's </span><span style="vertical-align: baseline;">tcp_ack()</span><span style="vertical-align: baseline;"> function was called to process it. And here, in kernel version 6.1 (which underpinned the </span><a href="https://docs.cloud.google.com/container-optimized-os/docs"><span style="text-decoration: underline; vertical-align: baseline;">Container-Optimized OS</span></a><span style="vertical-align: baseline;"> (COS) running on the KCP VMs), there was a bug.</span></p>
<p><span style="vertical-align: baseline;">RFC 5961 added stricter validation for acceptable ACK numbers — specifically, an ACK could be considered valid even if it was slightly "old" (within a sequence number window), to handle network reordering. But in kernel 6.1, there was a code path where </span><span style="vertical-align: baseline;">tcp_ack()</span><span style="vertical-align: baseline;"> returned zero — indicating "this packet was not fully processed" — when it encountered an ACK that was old but within the RFC 5961 acceptable window. The caller of </span><span style="vertical-align: baseline;">tcp_ack()</span><span style="vertical-align: baseline;"> in the FIN processing path checked this return value, saw zero, and did… nothing. The FIN was received and silently discarded.</span></p>
<p><strong style="vertical-align: baseline;">The Linux kernel commit that fixed it (upstream)</strong></p></div>
<div class="block-code"><dl>
<dt>code_block</dt>
<dd><ListValue: [StructValue([('code', 'commit 795a7dfbc3d9\r\nAuthor: Eric Dumazet <edumazet@google.com>\r\nnet: tcp: accept old ack during closing\r\n\r\ntcp_ack() returns 0 when ack is old but acceptable wrt rfc5961.\r\nIn FIN_WAIT_2, this caused tcp_fin() to never be called for a\r\nvalid received FIN, leaving the connection stuck indefinitely.\r\n\r\nFixed: properly handle old-but-acceptable ACKs during connection close.\r\nIntroduced in 6.1, fixed in 6.9.7. LTS backports: 6.6.y and 6.1.y.'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fcfb12a2f50>)])]></dd>
</dl></div>
<div class="block-paragraph_advanced"><p><span style="vertical-align: baseline;">The bug had existed since kernel 6.1 and was already fixed in kernel 6.9.7 — but GKE's KCP VMs were running COS builds based on 6.1 and 6.6. The fix needed to be backported and shipped as a COS image update.</span></p>
<h4><span style="vertical-align: baseline;">How the three bugs combined</span></h4>
<p><span style="vertical-align: baseline;">Individually, none of these three issues would have caused an outage of this magnitude. It took all three acting together to create the failure:</span></p>
<ul>
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
<p role="presentation"><strong style="vertical-align: baseline;">Bug #1 alone:</strong><span style="vertical-align: baseline;"> Packet reordering is mildly disruptive, but the kernel would normally still eventually process the FIN and clean up, just slower than ideal.</span></p>
</li>
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
<p role="presentation"><strong style="vertical-align: baseline;">Bug #2 alone: </strong><span style="vertical-align: baseline;">Envoy's half-close is perfectly correct behavior. It only becomes a problem if the FIN is never properly processed.</span></p>
</li>
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
<p role="presentation"><strong style="vertical-align: baseline;">Bug #3 alone:</strong><span style="vertical-align: baseline;"> The kernel's failure to process old-but-acceptable ACKs would cause some connections to linger, but the impact would be limited without the port-reuse pressure from Private Service Connect NAT.</span></p>
</li>
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
<p role="presentation"><strong style="vertical-align: baseline;">All three together:</strong><span style="vertical-align: baseline;"> Reordering causes the FIN to arrive out of sequence; the kernel bug prevents it from being processed; Envoy's SHUT_WR prevents the timeout from firing; the socket stays in FIN_WAIT_2 indefinitely. Private Service Connect NAT reuses the port. challenge ACK is sent. The USPS fix drops it. Connection fails.</span></p>
</li>
</ul>
<h4><span style="vertical-align: baseline;">The resolution: A three-pronged fix</span></h4>
<p><span style="vertical-align: baseline;">Resolving this required action at three different levels, coordinated across multiple teams over several weeks:</span></p>
<ul>
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
<p role="presentation"><strong style="vertical-align: baseline;">Immediate mitigation (Feb, 2026): </strong><span style="vertical-align: baseline;">Rollback of change in USPS. Restored the previous behavior where challenge ACKs were forwarded rather than dropped. Customer clusters recovered by early March.</span></p>
</li>
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
<p role="presentation"><strong style="vertical-align: baseline;">Correctness fix:</strong><span style="vertical-align: baseline;"> A replacement change to handle the challenge ACK scenario properly in the Private Service Connect proxy protocol layer, rather than simply restoring the old drop behavior. A PacketDrill test was added to the test suite to prevent regression.</span></p>
</li>
<li aria-level="1" style="list-style-type: disc; vertical-align: baseline;">
<p role="presentation"><strong style="vertical-align: baseline;">Root cause fix — Linux kernel patch:</strong><span style="vertical-align: baseline;"> Upstream patch ("net: tcp: accept old ack during closing"), authored by Eric Dumazet. Merged into Linux 6.9.7. Backported to 6.6.y and 6.1.y LTS branches. — shipping in GKE 1.33.5 and later. The bug is fully resolved at the OS level for all GKE versions running these COS images.</span></p>
</li>
</ul>
<h3><strong style="vertical-align: baseline;">What made this case different</strong></h3>
<p><span style="vertical-align: baseline;">Most networking cases resolve at one layer. You find the misconfigured firewall rule, the exhausted connection pool, the miscalculated MTU. You fix it and move on. This one was different because the symptom — </span><span style="vertical-align: baseline;">i/o timeout</span><span style="vertical-align: baseline;"> on a connection that never completed — was the same whether you were looking at the Private Service Connect layer, the kernel's TCP state machine, or Envoy's connection management. Every layer had a plausible explanation, and every explanation was incomplete.</span></p>
<p><span style="vertical-align: baseline;">What finally cracked it was the discipline of simultaneous packet capture at multiple points in the path. Without captures at the host NIC, the PIGv, and inside the container at the same time, we would never have seen the reordering. Without correlating those pcaps to the USPS metric spike on February 18, we would never have found the dropped challenge ACKs. And without reproducing the issue in a lab environment, we would never have found the kernel bug that made the FIN_WAIT_2 state permanent.</span></p>
<p><span style="vertical-align: baseline;">The other thing that made this work was the team. Over the course of two weeks, engineers from seven or eight different specialties contributed: GKE networking, Private Service Connect infrastructure, Hostnet, Andromeda, KCP, and ultimately the kernel community. No single person could have held all of this in their head. The case notes, the shared bug tracker, the meticulously collected pcaps — they were the connective tissue that let a distributed team work as one.</span></p>
<p><span style="vertical-align: baseline;">In the end, the customer got back to production. The kernel got a better bug fix. And the rest of us got a very good story about why packet reordering, half-closed sockets, and a well-intentioned security patch should never be allowed in the same room at the same time.</span></p>
<hr/>
<p><sub><span style="font-style: italic; vertical-align: baseline;">This blog is based on real case data from an anonymized customer support engagement and internal bugs. The Linux kernel fix referenced is commit </span><span style="font-style: italic; vertical-align: baseline;">795a7dfbc3d9</span><span style="font-style: italic; vertical-align: baseline;">, authored by Eric Dumazet. Special thanks to the engineers across GKE, Private Service Connect, Hostnet, Andromeda, COS, and the Linux networking community whose work made this resolution possible.</span></sub></p></div>