[CLUE-Tech] Linux TCP sockets SYN -> long delay waiting for tcp_syn_retries

[Jim Ockers]

Hi Dave,

Dude, you are The Man.  Thanks for the clue, something you wrote 
helped me to figure out the problem.

David Anselmi wrote:
> 
> Jim Ockers wrote:
> [...]
> > Actually the problem is not with telnet, it's with a Linux JMS client,
> 
> Not sure I have any ideas about that.
> 
> [...]
> >>What's the time between SYNs (all the same or using a back-off algorithm)?
> > 
> > 
> > I'm glad you asked - I think this is important.  Here is what ethereal
> > says about the timing for Linux telnet to a closed port:
> > 
> > time=0s	SYN sent (initial)
> > 1.4	RST received
> > 3.0	SYN sent, retry 1
> > 4.4	RST
> > 9.0	SYN sent, retry 2
> > 9.9	RST
> > 21.0	SYN sent, retry 3
> > 22.4	RST
> > 45.0	SYN sent, retry 4
> > 45.9	RST
> > 93.0	SYN sent, retry 5
> > 94.8	RST received
> > 194.5	[telnet process exits (this is not shown in ethereal of course)]
> 
> Interesting.  I get this behavior connecting to a blocked port (i.e., a 
> DROP rule in iptables--no RSTs sent).  And this is typical TCP behavior. 
>    The initial RTO (retransmission timeout) is 3 sec.  Each retry waits 
> twice as long as the first (3s, 6s, 12s, 24s, 48s, 96s).  The ~100s 
> delay at the end is waiting for a response from the final retry and 
> seems to be >180s by RFC (according to tcp.h).
> 
> If I telnet to a closed port and get a RST, only one SYN is sent, no 
> retries.  So it looks like your RSTs aren't matching the SYNs you send. 
>   Or maybe are blocked from getting to the sending machine (an INPUT 
> rule maybe?)

This was the key.  You are right, the RST that comes back doesn't match 
the SYN that was sent.  It turns out that there is an ACK bit that is 
set in the response from the server - ethereal shows it as [RST, ACK]
in the packet trace.  (The TCP flags bitmask is 0x14 for RST and ACK 
both set; the flags are 0x4 for just RST.)

The LAN and our "good" VSAT network both leave the ACK bit set, but
the "bad" VSAT system somehow strips out the ACK bit.  I think this
makes the Linux TCP/IP stack ignore the RST, as you surmised.  I
observed that if the ACK is present then the Linux TCP/IP processes 
it properly.

> Could it be that the VSAT mangles the RST so that it isn't recognized as 
> going with the SYN?  Or you're inadvertently blocking replies from the 
> VSAT IP or something.

The VSAT mangles the RST to strip off the ACK.  Thanks again for the
clue.  We are going to complain to the VSAT provider about this, and
hopefully they can fix it.

Thanks again,
Jim

-- 
Jim Ockers, P.Eng. (ockers at ockers.net)
Contact info: please see http://www.ockers.net/