<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
Hi everyone,<br>
<br>
Well to answer my own question: The reason iptables might not
masquerade some packets is because the TCP connection state was INVALID
because one of the connection tracker (ip_conntrack) timeouts was too
short. Once the timeout expires the connection tracker forgets
anything to do with that connection and the state becomes invalid, so
any further packets belonging to that connection are not handled by
netfilter in any kind of sane way. In this case they would not be
MASQUERADEd and could leak out with an invalid internal source IP
address.<br>
<br>
The workaround is two-fold:<br>
1. Make sure iptables will DROP any INVALID state packets.<br>
$IPTABLES -A FORWARD -i $INTIF -p tcp -m state --state INVALID -j
DROP<br>
<br>
2. Increase the timeout for CLOSE_WAIT state in the connection tracker.<br>
echo 3600
>/proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_close_wait<br>
<br>
Having made those 2 changes I have observed that we do not get
disconnected from Verizon anymore which means that they are not seeing
many (if any) non-NATted packets anymore. There is an excellent
writeup about this problem here:
<a class="moz-txt-link-freetext" href="http://www.smythies.com/~doug/network/iptables_notes/index.html">http://www.smythies.com/~doug/network/iptables_notes/index.html</a> He has
also submitted this as a bug to the netfilter team, but I can imagine a
fair bit of blame-storming over this one because you can't really
expect netfilter to handle invalid packets or packets that don't belong
to any known connection in any kind of sane way.<br>
<br>
Jim<br>
-- <br>
Jim Ockers, P.Eng. (<a class="moz-txt-link-abbreviated" href="mailto:ockers@ockers.net">ockers@ockers.net</a>)<br>
Contact info: <a class="moz-txt-link-freetext" href="http://www.ockers.net/msi.html">http://www.ockers.net/msi.html</a><br>
<br>
<br>
Jim Ockers wrote:
<blockquote cite="mid:4E1C1D71.1010603@ockers.net" type="cite"><font
size="+1"><tt>Hi CLUEbies,<br>
<br>
As usual when I can't figure something out I like to ask the list. I
have a Linux 2.6.32.27 (armv5teb - Intel XScale) system which acts as a
router and has a variety of network connections including 3G cellular.
The linux firewall is configured to masquerade internal network traffic
so that it should appear to come from the router's own outside
interface. Unfortunately for some reason some packets aren't NATted
and manage to escape through the PPP interface still bearing the
(invalid) internal IP address.<br>
<br>
When I use a 3G PPP interface as the default gateway, and the 3G data
provider happens to be Verizon, then Verizon's network security (which
has a threshold of invalid packets/time) disconnects the PPP connection
by sending an LCP Termination Request. Normally this would not be a
problem but somehow this router/firewall are letting some packets sneak
through without NATting them. I attached a screen shot of the
offending packets, which are from a PPP record file created by the
"record /tmp/filename" option and which wireshark knows how to decode.
The recorded file contains all bits written to or received from the
serial port during the course of the PPP daemon running, and so there
should never be any non-NATted packets show up in this file. The
screen shot shows a few and this was enough to get Verizon to
disconnect.<br>
<br>
Since I can't attach files and send them to the list here is the screen
shot: <a moz-do-not-send="true" class="moz-txt-link-freetext"
href="http://chile.mudlogsys.com/%7Eockers/wireshark.png">http://chile.mudlogsys.com/~ockers/wireshark.png</a><br>
<IMG SRC=<a moz-do-not-send="true" class="moz-txt-link-rfc2396E"
href="http://chile.mudlogsys.com/%7Eockers/wireshark.png">"http://chile.mudlogsys.com/~ockers/wireshark.png"</a>><br>
<br>
The relevant fine print & details:<br>
- The default route is the PPP interface device, no gateway specified<br>
- There is no iproute2 or other routing funny business<br>
- Some of the other iptables prerouting/etc rules are a bit wonky but
the nat/POSTROUTING and mangle/POSTROUTING are quite clear - see below<br>
- The raw table has nothing in it<br>
- We ARE doing traffic shaping using u32 classifier and imq0. All
packets are looped through the IMQ device for htb prioritization.<br>
- The source IP address in the packets as shown in the screen shot is
a garden variety Apple macbook laptop.<br>
- Note that every one of the un-masqueraded packets in this capture
are TLSv1 to port
993 (IMAP/S). I think this is simply a co-incidence because I've seen
other types of traffic in other disconnect tests including plain old
HTTP.<br>
- I have a second screen shot showing all 993/tcp packets. 3 of
the 16 packets are properly NATted. The rest are not NATted.<br>
- AT&T just discards improperly addressed packets. Verizon hangs
up the connection. I can see that it might be nice to keep "bad"
traffic off their network so I can't object to this policy.<br>
<br>
<a moz-do-not-send="true" class="moz-txt-link-freetext"
href="http://chile.mudlogsys.com/%7Eockers/wireshark-imaps.png">http://chile.mudlogsys.com/~ockers/wireshark-imaps.png</a><br>
<IMG SRC=<a moz-do-not-send="true" class="moz-txt-link-rfc2396E"
href="http://chile.mudlogsys.com/%7Eockers/wireshark-imaps.png">"http://chile.mudlogsys.com/~ockers/wireshark-imaps.png"</a>><br>
<br>
root@Router:~# iptables -t nat -L POSTROUTING -n -v <br>
Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)<br>
pkts bytes target prot opt in out source
destination <br>
0 0 ACCEPT all -- * * 10.0.0.0/8
10.0.0.0/8 <br>
0 0 ACCEPT all -- * * 192.168.0.0/16
192.168.0.0/16 <br>
0 0 MASQUERADE all -- * * 0.0.0.0/0
0.0.0.0/0 <br>
root@Router:~# iptables -t mangle -L POSTROUTING -n -v<br>
Chain POSTROUTING (policy ACCEPT 488K packets, 353M bytes)<br>
pkts bytes target prot opt in out source
destination <br>
root@Router:~#<br>
<br>
The counters are zeroed out each time it reconnects, so don't mind the
zero pkt/byte counters there.<br>
<br>
I don't really know what else to check for. Given that catch-all
MASQUERADE rule, how could any of these packets sneak past it without
getting masqueraded?<br>
<br>
Thanks for any advice or suggestions,<br>
Jim<br>
</tt></font>
<pre class="moz-signature" cols="72">--
Jim Ockers, P.Eng. (<a moz-do-not-send="true"
class="moz-txt-link-abbreviated" href="mailto:ockers@ockers.net">ockers@ockers.net</a>)
Contact info: <a moz-do-not-send="true" class="moz-txt-link-freetext"
href="http://www.ockers.net/msi.html">http://www.ockers.net/msi.html</a>
</pre>
</blockquote>
</body>
</html>