[clue] [tech] why would iptables not masquerade some IMAP/S (993/tcp) packets

[Jim Ockers]

Hi everyone,

Well to answer my own question: The reason iptables might not masquerade 
some packets is because the TCP connection state was INVALID because one 
of the connection tracker (ip_conntrack) timeouts was too short.  Once 
the timeout expires the connection tracker forgets anything to do with 
that connection and the state becomes invalid, so any further packets 
belonging to that connection are not handled by netfilter in any kind of 
sane way.  In this case they would not be MASQUERADEd and could leak out 
with an invalid internal source IP address.

The workaround is two-fold:
1. Make sure iptables will DROP any INVALID state packets.
    $IPTABLES -A FORWARD -i $INTIF -p tcp -m state --state INVALID -j DROP

2. Increase the timeout for CLOSE_WAIT state in the connection tracker.
    echo 3600 
 >/proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_timeout_close_wait

Having made those 2 changes I have observed that we do not get 
disconnected from Verizon anymore which means that they are not seeing 
many (if any) non-NATted packets anymore.  There is an excellent writeup 
about this problem here: 
http://www.smythies.com/~doug/network/iptables_notes/index.html  He has 
also submitted this as a bug to the netfilter team, but I can imagine a 
fair bit of blame-storming over this one because you can't really expect 
netfilter to handle invalid packets or packets that don't belong to any 
known connection in any kind of sane way.

Jim
-- 
Jim Ockers, P.Eng. (ockers at ockers.net)
Contact info: http://www.ockers.net/msi.html


Jim Ockers wrote:
> Hi CLUEbies,
>
> As usual when I can't figure something out I like to ask the list.  I 
> have a Linux 2.6.32.27 (armv5teb - Intel XScale) system which acts as 
> a router and has a variety of network connections including 3G 
> cellular.  The linux firewall is configured to masquerade internal 
> network traffic so that it should appear to come from the router's own 
> outside interface.  Unfortunately for some reason some packets aren't 
> NATted and manage to escape through the PPP interface still bearing 
> the (invalid) internal IP address.
>
> When I use a 3G PPP interface as the default gateway, and the 3G data 
> provider happens to be Verizon, then Verizon's network security (which 
> has a threshold of invalid packets/time) disconnects the PPP 
> connection by sending an LCP Termination Request.  Normally this would 
> not be a problem but somehow this router/firewall are letting some 
> packets sneak through without NATting them.  I attached a screen shot 
> of the offending packets, which are from a PPP record file created by 
> the "record /tmp/filename" option and which wireshark knows how to 
> decode.  The recorded file contains all bits written to or received 
> from the serial port during the course of the PPP daemon running, and 
> so there should never be any non-NATted packets show up in this file.  
> The screen shot shows a few and this was enough to get Verizon to 
> disconnect.
>
> Since I can't attach files and send them to the list here is the 
> screen shot: http://chile.mudlogsys.com/~ockers/wireshark.png
> <IMG SRC="http://chile.mudlogsys.com/~ockers/wireshark.png">
>
> The relevant fine print & details:
>  - The default route is the PPP interface device, no gateway specified
>  - There is no iproute2 or other routing funny business
>  - Some of the other iptables prerouting/etc rules are a bit wonky but 
> the nat/POSTROUTING and mangle/POSTROUTING are quite clear - see below
>  - The raw table has nothing in it
>  - We ARE doing traffic shaping using u32 classifier and imq0.  All 
> packets are looped through the IMQ device for htb prioritization.
>  - The source IP address in the packets as shown in the screen shot is 
> a garden variety Apple macbook laptop.
>  - Note that every one of the un-masqueraded packets in this capture 
> are TLSv1 to port 993 (IMAP/S).  I think this is simply a co-incidence 
> because I've seen other types of traffic in other disconnect tests 
> including plain old HTTP.
>  - I have a second screen shot showing all 993/tcp packets.  3 of the 
> 16 packets are properly NATted.  The rest are not NATted.
>  - AT&T just discards improperly addressed packets.  Verizon hangs up 
> the connection.  I can see that it might be nice to keep "bad" traffic 
> off their network so I can't object to this policy.
>
> http://chile.mudlogsys.com/~ockers/wireshark-imaps.png
> <IMG SRC="http://chile.mudlogsys.com/~ockers/wireshark-imaps.png">
>
> root at Router:~# iptables -t nat -L POSTROUTING -n -v
> Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
>  pkts bytes target     prot opt in     out     source               
> destination        
>     0     0 ACCEPT     all  --  *      *       10.0.0.0/8           
> 10.0.0.0/8         
>     0     0 ACCEPT     all  --  *      *       192.168.0.0/16       
> 192.168.0.0/16     
>     0     0 MASQUERADE  all  --  *      *       0.0.0.0/0            
> 0.0.0.0/0          
> root at Router:~# iptables -t mangle -L POSTROUTING -n -v
> Chain POSTROUTING (policy ACCEPT 488K packets, 353M bytes)
>  pkts bytes target     prot opt in     out     source               
> destination        
> root at Router:~#
>
> The counters are zeroed out each time it reconnects, so don't mind the 
> zero pkt/byte counters there.
>
> I don't really know what else to check for.  Given that catch-all 
> MASQUERADE rule, how could any of these packets sneak past it without 
> getting masqueraded?
>
> Thanks for any advice or suggestions,
> Jim
> -- 
> Jim Ockers, P.Eng. (ockers at ockers.net)
> Contact info: http://www.ockers.net/msi.html
>
>   
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://cluedenver.org/pipermail/clue/attachments/20110712/68147cd1/attachment.html