About the TCP MSS and wrong checksums

I spent a few hours troubleshooting a strange TCP connection problem. HTTP connections to one particular host were no longer working since we got a new Cisco ASA 5510 firewall managed by an external provider, set up to NAT outgoing connections. Initial connections to that host and the first HTTP request returning a redirect worked, but the following request would only return a few KB of HTML and then stall. Only that host was affected, at least we had not observed the problem somewhere else.

I did a tcpdump of the traffic between the firewall and our SDSL uplink and got the following.

mp@blackbook:~$ sudo tcpdump -v -ttt -s0 -n host remote.host
tcpdump: listening on en0, link-type EN10MB (Ethernet), capture size 65535 bytes
000000 IP ([...], length 48) firewall.26167 > remote.host.80: S, cksum 0x0b7b (correct), 3325973421:3325973421(0) win 65535 <mss 1380,nop,nop,nop,nop>
213304 IP ([...], length 44) remote.host.80 > firewall.26167: S, cksum 0xe020 (correct), 83736657:83736657(0) ack 3325973422 win 32768 <mss 1380>
000320 IP ([...], length 40) firewall.26167 > remote.host.80: ., cksum 0x778e (correct), ack 1 win 65535
000555 IP ([...], length 396) firewall.26167 > remote.host.80: P, cksum 0x3f23 (correct), 1:357(356) ack 1 win 65535
225193 IP ([...], length 317) remote.host.80 > firewall.26167: P, cksum 0xdd51 (correct), 1:278(277) ack 357 win 32768
017471 IP ([...], length 1420) remote.host.80 > firewall.26167: P, cksum 0x9fce (incorrect (-> 0x9fcd), 278:1658(1380) ack 357 win 32768
016391 IP ([...], length 1420) remote.host.80 > firewall.26167: P, cksum 0x766d (incorrect (-> 0x766c), 1658:3038(1380) ack 357 win 32768
203816 IP ([...], length 40) firewall.26167 > remote.host.80: ., cksum 0x762a (correct), ack 278 win 65258
226371 IP ([...], length 1420) remote.host.80 > firewall.26167: ., cksum 0x6120 (incorrect (-> 0x611f), 3038:4418(1380) ack 357 win 32768
2. 398071 IP ([...], length 1420) remote.host.80 > firewall.26167: P, cksum 0x9fce (incorrect (-> 0x9fcd), 278:1658(1380) ack 357 win 32768
2. 947090 IP ([...], length 1420) remote.host.80 > firewall.26167: P, cksum 0x9fce (incorrect (-> 0x9fcd), 278:1658(1380) ack 357 win 32768
5. 479496 IP ([...], length 1420) remote.host.80 > firewall.26167: P, cksum 0x9fce (incorrect (-> 0x9fcd), 278:1658(1380) ack 357 win 32768

That looked as if the remote host was sending a lot of packets with wrong TCP checksums. Our firewall would just trash those packets, not sending out ACKs for them and wait for a retransmission. Those retransmissions can be seen at the end of the trace and trickled away with geometrically increasing intervals. As the retransmissions show the same error, the bottom line is that nothing gets along anymore.

Interestingly, the checksums were always off by just one. Had the packets really been altered (for example, corrupted by bit errors on the wire) during transport, the deviations should be arbitrary.

So my first guess was that the remote host or some component along the path had a bug in the procotol stack and was putting in incorrectly calculated checksums. But OTOH it's hard to imagine that we would be the first to discover such a bug.

When taking the firewall out of the mix the connections worked fine and all packets were ok. This is strange, as the TCP checksum is calculated based on a single TCP packet with some additional information from the IP layer. So as to the correctness and verification of the checksum, it is irrelevant what has been sent in the other direction before and whether such previous packets have passed the firewall or not.

Now one important observation was that the incorrect checksum was only in packets that were 1420 bytes in total size. This is the maximum possible size one can observe in this connection for a MSS of 1380 is negotiated during the TCP handshake and 40 byte for IP and TCP header come on top of that.

Using a test host placed outside (before) the firewall would result in working connections with an MSS of 1460, which is the Ethernet MTU of 1500 minus the 40 bytes. Setting the MTU to 1420 on that host yielded a MSS of 1380, but the returning packets were correctly checksummed in this case.

So although the MSS seemed not to be a sufficient condition to trigger the problem, I started to investigate why the firewall would use a value of 1380 where 1460 should be possible. We found a setting called "force maximum segment size for TCP proxy connections" in the Cisco's admin interface that was set to 1380. We unset that option and voilà, everything worked with the MSS going up to 1460.

What rankles me is that although the symptoms are cured, I still haven't found the cause for the wrong checksums. I don't see why setting the MSS to a lower value than necessary should be a problem at all (performance issues aside). And, having the checksums always off-by-one smells like another bug somewhere along the connection's path. I googled a lot for this one, but the only relevant results were in a discussion in the context of SMTP (but that's TCP after all) over there and there.

Hopefully, this article will save someone some headaches. If you have any ideas regarding the cause of this problem, I'd be glad to hear from you. Feel free to leave a comment!