Welcome to End Point’s blog

Ongoing observations by End Point people

Cisco PIX mangled packets and iptables state tracking

Kiel and I had a fun time tracking down a client's networking problem the other day. Their scp transfers from their application servers behind a Cisco PIX firewall failed after a few seconds, consistently, with a connection reset.

The problem was easily reproducible with packet sizes of 993 bytes or more, not just with TCP but also ICMP (bloated ping packets, generated with ping -s 993 $host). That raised the question of how this problem could go undetected for their heavy web traffic. We determined that their HTTP load balancer avoided the problem as it rewrote the packets for HTTP traffic on each side.

Kiel narrowed the connect resets down to iptables' state-tracking considering packets INVALID, not ESTABLISHED or RELATED as they should be.

Then he found via tcpdump that the problem was easily visible in scp connections when TCP window scaling adjustments were made by either side of the connection. We tried disabling window scaling but that didn't help.

We tried having iptables allow packets in state INVALID when they were also ESTABLISHED or RELATED, and that reduced the frequency of terminated connections, but still didn't eliminate them entirely. (And it was a kludge we weren't eager to keep in place anyway.)

We wanted to avoid some unpleasant possibilities: (1) turn off stateful firewalling or (2) perform risky updates or configuration changes on the Cisco PIX, which may or may not fix the problem, in the middle of the busy holiday ecommerce season.

Finally, Kiel found this netfilter mailing list post which describes how to enable a Linux kernel workaround for the mangled packets the Cisco generates:

echo 1 > /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_be_liberal

Of course saving that in /etc/sysctl.conf so it persists after a reboot.

So we have reliable long-running scp connections with TCP window scaling working and iptables doing its job. I love it when a plan comes together.


Pat Morin said...

Woah. This finally resolved a problem we were having with an otherwise flawless Linux firewall box I've been using in my lab.

Thanks for putting this out there.

Jon Jensen said...

Glad to hear it helped, Pat!

Big dog said...

Do you know if this has any impact on security, in terms of invalid packet attacks? This 'be liberal' sounds like it allows certain types of invalid packets.

Also I think this should be fixed on the cisco, thats the box giving out the 'invalid' packets? (mind you I have zero experience with cisco boxes, its easier to say than do ;) ).

Thanks for the post though, you've put it in easy to understand terms!

Jon Jensen said...

I'm sure it must be allowing some invalid packets, though I don't know the details. I'm not very worried about it because the Linux networking stack probably just ignores them.

Yes, it should be fixed on the Cisco box, but good luck getting Cisco to do that. Apparently others have tried. :)