<div dir="ltr"><div><div><div><div><div><div>Sorry I didn't get a chance to respond to this, I wanted to take a second and say thank you to everybody who's provided feedback. I'm sorry I didn't respond to it sooner.<br>
<br></div>I had an old co-worker, who used to say, follow the chain. So here's the chain. A RHEL 6.3 box (A), living on an ESXi 5.0.0 build-623860 box, with an eth0 set to 9000 mtu is connected to a vswitch within the host, set to 9000 mtu, which is connected to a 10GB physical wire. Next comes uncharted network space, which I have no access to, but our networking guys assure me six ways from sunday contains no relevant speed or port controls. Followed by an VMware ESXi 5.1.0 build-1065491 host, running on a physical 10GB wire, with a vswitch <br>
</div>set to 9000mtu, and a RHEL6.3 box set to 9000 mtu in the OS. So the chain looks like<br><br></div>RHEL6.3 (A) -> vswitch -> ESXi5.0 -> 10GB wire -> cloud<br></div>9000 mtu 9000<br> <br></div>cloud -> 10 GB wire -> ESXi5.1 -> vswitch -> RHEL6.3(B)<br>
</div> 9000 mtu 9000 mtu<br><div><div><br></div><div>B can SCP to A, but A cannot SCP to B<br></div><div>Set A to 1500 mtu, and now A CAN scp to B<br></div><div>Set A back to 9000, and scp breaks again.<br>
<br></div><div>Set B to 1500 and A to 9000, and it works again<br></div><div>So it works if one is 1500, and the other 9000, but it doesn't matter which one, they just can't both be 9000. Because I'm kinda the defacto vmware guy on the unix team I work on, and because I'm running out of ideas, I literally exported A from the 5.0 host that it was living on, imported it into the 5.1 host that B was living on, and tried again with the same virtual machine, totally identical except for it's IP address. (I had to change A's IP address because A and B live in different subnets.)<br>
And it worked! (scratches head)<br><br></div><div>Unfortunately, I can't leave it on the wrong subnet, that's not a good long term option. I havent' read some of the feedback in this thread yet, so I'm going to go back and read some of the replies, but right now, I've got zero ideas.<br>
<br></div><div>Mike Bean<br></div></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Oct 15, 2013 at 5:37 AM, Jim Bucks <span dir="ltr"><<a href="mailto:jbucks@procci.com" target="_blank">jbucks@procci.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>This is a GREAT explanation!<br>
<br>
I've had some personal experience setting these up, and have had
more then a few "head scratcing" moments, too.<br>
<br>
The worst one was (all 10 Gig MM Fiber):<br>
<font face="Courier New, Courier, monospace"> Sun NAS (4541 -
Thumper) <-> HP Procurve Switch <-> Isilon NAS<br>
<->
Windows XP Workstations<br>
</font><br>
<br>
No matter what I did, the only way I could get this to work
reliably (and a config that should NOT have worked) was to set the
Switch, Isilon, and Sun NAS to use Jumbo Frames and to set the
Windows XP W/S's to standard Frame sizes (1500). Any other
combination resulted in laboriosly slow file (gargantuan HD Video
Files) transfers / access.<br>
<br>
Oh, and tried the file access via both SMB/CIFS and NFS no real
difference in ether one.<br>
<br>
Sometimes you have to just do what works in spite of whether it
should (or not).<br>
<br>
Jim<div><div class="h5"><br>
<br>
<br>
<br>
<br>
<br>
On 10/14/2013 08:25 PM, Mark G. Harvey wrote:<br>
</div></div></div><div><div class="h5">
<blockquote type="cite">
<div style="font-size:8pt;font-family:HelveticaNeue,Helvetica Neue,Helvetica,Arial,Lucida Grande,sans-serif">Having worked tech
support on several of jumbo frames calls, here's what I learned
<br>
<br>
To cover the overhead of Jumbo Frames, the switch & its
ports have to be set at a higher MTU to pass the additional
headers added when a packet leaves eth0 Important: every
interface that handles a packet needs to be set for Jumbo
Frames. At the hosts, 9000 ... at the switch globally & at
each port 9216 <br>
<br>
HostOne<br>
eth0 w/ MTU=9000<br>
| <br>
cable<br>
| <br>
switch w/ MTU=9216 set globally PLUS <br>
MTU=9216 set at specific ports passing Jumbo Frames<br>
| <br>
cable<br>
| <br>
HostTwo<br>
eth0 w/ MTU=9000 <br>
<br>
Given that it works in your system one way, but not the reverse,
there could be just one switch port that still has the default
MTU=1500 <br>
<br>
Google this: jumbo frames switch mtu 9216 ... a lot of good
info there<br>
<br>
<br>
Best of luck on solving the problem. <br>
<br>
<div><span></span></div>
<div style="display:block"> <br>
<br>
<div style="font-family:HelveticaNeue,Helvetica Neue,Helvetica,Arial,Lucida Grande,sans-serif;font-size:8pt">
<div style="font-family:HelveticaNeue,Helvetica Neue,Helvetica,Arial,Lucida Grande,sans-serif;font-size:12pt">
<div dir="ltr"> <font face="Arial"> On Monday,
October 14, 2013 4:37 PM, Quentin Hartman
<a href="mailto:qhartman@gmail.com" target="_blank"><qhartman@gmail.com></a> wrote:<br>
</font> </div>
<div>
<div>
<div>
<div dir="ltr">Well, what do the logical groupings
of servers have to do with the network topology?
What exactly lies between server A and B? Are
there any weird tunnels? Do all the switches
support jumbo frames? All the routers correctly
translating between networks?
<div>
<br clear="none">
</div>
<div>MTU is a fiddly thing, and if every device in
the path isn't correctly configured you will get
weird behavior.</div>
<div><br clear="none">
</div>
<div>QH</div>
</div>
<div><br clear="none">
<br clear="none">
<div>
<div>
On Mon, Oct 14, 2013 at 3:57 PM, Mike Bean <span dir="ltr"><<a rel="nofollow" shape="rect" href="mailto:beandaemon@gmail.com" target="_blank">beandaemon@gmail.com</a>></span>
wrote:<br clear="none">
<blockquote style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div>
<div>
<div>Well, I don't mean OU in the
windows sense, I mean logical
groupings of servers. Clusters.
I'm trying to get my head around
this, because intuitively, it just
doesn't make any sense. <br clear="none">
<br clear="none">
</div>
If you're coming home from the grocery
store, and you have more groceries
then you can carry, you make multiple
trips to the car. <br clear="none">
<br clear="none">
</div>
but I don't think it's quite that
simple. Think networks aren't as smart
as I think they are, intuitively,
something configured to send as 1500,
should be able to receive 1500 only, not
necessary 9000, but something that's
configured as 9000 should be able to
send and receive as 9000 or 1500.
(scratches head.)<br clear="none">
<br clear="none">
</div>
<div>In actuality, B (9000) can send to A
(1500)<br clear="none">
</div>
<div>and A (1500) can send to B (9000)<br clear="none">
<br clear="none">
</div>
<div>but if we set A to 9000, it can no
longer send to B (9000)<br clear="none">
</div>
<div>(scratches head)<br clear="none">
</div>
<div><br clear="none">
</div>
Boy, if you know someone who's in school
looking for a secure job, tell them to
become a network person. REALLY GOOD
network people are worth their weight in
gold.<br clear="none">
</div>
<div>
<div>
<div><br clear="none">
<br clear="none">
<div>On
Mon, Oct 14, 2013 at 3:26 PM,
Quentin Hartman <span dir="ltr"><<a rel="nofollow" shape="rect" href="mailto:qhartman@gmail.com" target="_blank">qhartman@gmail.com</a>></span>
wrote:<br clear="none">
<blockquote style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">By OU, do you mean
broadcast domain? ie - two LANs
connected by a router? If so, it
sounds like you have the
(usually default) MTU of 1500
set on side A, but the 9000 set
on B, and your router isn't
correctly doing MTU management
when crossing from one to the
other.
<div>
<br clear="none">
</div>
<div>Going the other way works
because the fragments are
already small enough to be
handled correctly at the other
end. The complexity of making
this work reliably is the
primary reason so few people
bother to use jumbo frames,
even though they technically
are better most of the time
anymore.</div>
<div><br clear="none">
</div>
<div>The wikipedia article on
MTU is actually pretty good: <a rel="nofollow" shape="rect" href="http://en.wikipedia.org/wiki/Maximum_transmission_unit" target="_blank">http://en.wikipedia.org/wiki/Maximum_transmission_unit</a></div>
<div>
<br clear="none">
</div>
<div>
QH</div>
</div>
<div><br clear="none">
<br clear="none">
<div>
<div>
<div>On Mon, Oct 14, 2013 at
3:15 PM, Mike Bean <span dir="ltr"><<a rel="nofollow" shape="rect" href="mailto:beandaemon@gmail.com" target="_blank">beandaemon@gmail.com</a>></span>
wrote:<br clear="none">
</div>
</div>
<blockquote style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div>
<div dir="ltr">
<div>
<div>
<div>
<div>
<div><br clear="none">
</div>
So I've been
scratching my
head for a
long time
trying to
understand
this, and it
makes no sense
to me. We
have to
discrete
OU's. SCP
transmissions
of any real
size tend to
fail from A to
B, but not
from B to A.
<br clear="none">
<br clear="none">
</div>
My colleagues
have been
arguing that the
fix is to set
the MTU at side
B to 1500. And
Lo, and behold,
when we do, it
works, increase
the MTU to 9000,
and it fails
again.<br clear="none">
<br clear="none">
<a rel="nofollow" shape="rect" href="http://unix.stackexchange.com/questions/14187/why-does-scp-hang-on-copying-files-larger-than-1405-bytes" target="_blank">http://unix.stackexchange.com/questions/14187/why-does-scp-hang-on-copying-files-larger-than-1405-bytes</a><br clear="none">
<br clear="none">
</div>
What I have
difficulty
understanding, is
if a jumbo frame
carries, say, 9000
bytes, and
hypothetically, if
it's a 5,000 bytes
file + your MTU =
1500, which means
you split it up
into 3
transmissions.<br clear="none">
<br clear="none">
</div>
Same size
transmission; capped
packet size, more
packets. So I
would naturally
conclude that at an
MTU of 9000 it could
get done in 1 what
an MTU of 1500 would
do in 3?<br clear="none">
<br clear="none">
</div>
(scratches head)<br clear="none">
</div>
<br clear="none">
</div>
</div>
_______________________________________________<br clear="none">
clue mailing list: <a rel="nofollow" shape="rect" href="mailto:clue@cluedenver.org" target="_blank">clue@cluedenver.org</a><br clear="none">
For information, account
preferences, or to
unsubscribe see:<br clear="none">
<a rel="nofollow" shape="rect" href="http://cluedenver.org/mailman/listinfo/clue" target="_blank">http://cluedenver.org/mailman/listinfo/clue</a><br clear="none">
</blockquote>
</div>
<br clear="none">
</div>
<br clear="none">
_______________________________________________<br clear="none">
clue mailing list: <a rel="nofollow" shape="rect" href="mailto:clue@cluedenver.org" target="_blank">clue@cluedenver.org</a><br clear="none">
For information, account
preferences, or to unsubscribe
see:<br clear="none">
<a rel="nofollow" shape="rect" href="http://cluedenver.org/mailman/listinfo/clue" target="_blank">http://cluedenver.org/mailman/listinfo/clue</a><br clear="none">
</blockquote>
</div>
<br clear="none">
</div>
</div>
</div>
<br clear="none">
_______________________________________________<br clear="none">
clue mailing list: <a rel="nofollow" shape="rect" href="mailto:clue@cluedenver.org" target="_blank">clue@cluedenver.org</a><br clear="none">
For information, account preferences, or to
unsubscribe see:<br clear="none">
<a rel="nofollow" shape="rect" href="http://cluedenver.org/mailman/listinfo/clue" target="_blank">http://cluedenver.org/mailman/listinfo/clue</a><br clear="none">
</blockquote>
</div>
<br clear="none">
</div>
</div>
</div>
</div>
<br>
<div>_______________________________________________<br clear="none">
clue mailing list: <a shape="rect" href="mailto:clue@cluedenver.org" target="_blank">clue@cluedenver.org</a><br clear="none">
For information, account preferences, or to
unsubscribe see:<br clear="none">
<a shape="rect" href="http://cluedenver.org/mailman/listinfo/clue" target="_blank">http://cluedenver.org/mailman/listinfo/clue</a></div>
<br>
<br>
</div>
</div>
</div>
</div>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
clue mailing list: <a href="mailto:clue@cluedenver.org" target="_blank">clue@cluedenver.org</a>
For information, account preferences, or to unsubscribe see:
<a href="http://cluedenver.org/mailman/listinfo/clue" target="_blank">http://cluedenver.org/mailman/listinfo/clue</a></pre>
</blockquote>
<br>
<br>
</div></div><span class="HOEnZb"><font color="#888888"><pre cols="72">--
+--------------------------------------+------------------------------+
|Jim Bucks |Phone: +1(970)539-1242 Cell |
|1924 44th Avenue Court | +1(970)330-3276 Home |
|Greeley, Colorado 80634 (USA) |Email: <a href="mailto:jbucks@procci.com" target="_blank">jbucks@procci.com</a> |
+--------------------------------------+------------------------------+
| WWW: <a href="http://www.procci.com" target="_blank">http://www.procci.com</a> |
+---------------------------------------------------------------------+
</pre>
</font></span></div>
<br>_______________________________________________<br>
clue mailing list: <a href="mailto:clue@cluedenver.org">clue@cluedenver.org</a><br>
For information, account preferences, or to unsubscribe see:<br>
<a href="http://cluedenver.org/mailman/listinfo/clue" target="_blank">http://cluedenver.org/mailman/listinfo/clue</a><br></blockquote></div><br></div>