[clue] a networking riddle
Mike Bean
beandaemon at gmail.com
Wed Oct 16 12:37:51 MDT 2013
Sorry I didn't get a chance to respond to this, I wanted to take a second
and say thank you to everybody who's provided feedback. I'm sorry I
didn't respond to it sooner.
I had an old co-worker, who used to say, follow the chain. So here's the
chain. A RHEL 6.3 box (A), living on an ESXi 5.0.0 build-623860 box, with
an eth0 set to 9000 mtu is connected to a vswitch within the host, set to
9000 mtu, which is connected to a 10GB physical wire. Next comes uncharted
network space, which I have no access to, but our networking guys assure me
six ways from sunday contains no relevant speed or port controls.
Followed by an VMware ESXi 5.1.0 build-1065491 host, running on a physical
10GB wire, with a vswitch
set to 9000mtu, and a RHEL6.3 box set to 9000 mtu in the OS. So the chain
looks like
RHEL6.3 (A) -> vswitch -> ESXi5.0 -> 10GB wire -> cloud
9000 mtu 9000
cloud -> 10 GB wire -> ESXi5.1 -> vswitch -> RHEL6.3(B)
9000 mtu 9000 mtu
B can SCP to A, but A cannot SCP to B
Set A to 1500 mtu, and now A CAN scp to B
Set A back to 9000, and scp breaks again.
Set B to 1500 and A to 9000, and it works again
So it works if one is 1500, and the other 9000, but it doesn't matter which
one, they just can't both be 9000. Because I'm kinda the defacto vmware
guy on the unix team I work on, and because I'm running out of ideas, I
literally exported A from the 5.0 host that it was living on, imported it
into the 5.1 host that B was living on, and tried again with the same
virtual machine, totally identical except for it's IP address. (I had to
change A's IP address because A and B live in different subnets.)
And it worked! (scratches head)
Unfortunately, I can't leave it on the wrong subnet, that's not a good long
term option. I havent' read some of the feedback in this thread yet, so
I'm going to go back and read some of the replies, but right now, I've got
zero ideas.
Mike Bean
On Tue, Oct 15, 2013 at 5:37 AM, Jim Bucks <jbucks at procci.com> wrote:
> This is a GREAT explanation!
>
> I've had some personal experience setting these up, and have had more then
> a few "head scratcing" moments, too.
>
> The worst one was (all 10 Gig MM Fiber):
> Sun NAS (4541 - Thumper) <-> HP Procurve Switch <-> Isilon NAS
> <-> Windows XP
> Workstations
>
>
> No matter what I did, the only way I could get this to work reliably (and
> a config that should NOT have worked) was to set the Switch, Isilon, and
> Sun NAS to use Jumbo Frames and to set the Windows XP W/S's to standard
> Frame sizes (1500). Any other combination resulted in laboriosly slow file
> (gargantuan HD Video Files) transfers / access.
>
> Oh, and tried the file access via both SMB/CIFS and NFS no real difference
> in ether one.
>
> Sometimes you have to just do what works in spite of whether it should (or
> not).
>
> Jim
>
>
>
>
>
>
> On 10/14/2013 08:25 PM, Mark G. Harvey wrote:
>
> Having worked tech support on several of jumbo frames calls, here's what I
> learned
>
> To cover the overhead of Jumbo Frames, the switch & its ports have to be
> set at a higher MTU to pass the additional headers added when a packet
> leaves eth0 Important: every interface that handles a packet needs to be
> set for Jumbo Frames. At the hosts, 9000 ... at the switch globally & at
> each port 9216
>
> HostOne
> eth0 w/ MTU=9000
> |
> cable
> |
> switch w/ MTU=9216 set globally PLUS
> MTU=9216 set at specific ports passing Jumbo Frames
> |
> cable
> |
> HostTwo
> eth0 w/ MTU=9000
>
> Given that it works in your system one way, but not the reverse, there
> could be just one switch port that still has the default MTU=1500
>
> Google this: jumbo frames switch mtu 9216 ... a lot of good info there
>
>
> Best of luck on solving the problem.
>
>
>
> On Monday, October 14, 2013 4:37 PM, Quentin Hartman
> <qhartman at gmail.com> <qhartman at gmail.com> wrote:
> Well, what do the logical groupings of servers have to do with the
> network topology? What exactly lies between server A and B? Are there any
> weird tunnels? Do all the switches support jumbo frames? All the routers
> correctly translating between networks?
>
> MTU is a fiddly thing, and if every device in the path isn't correctly
> configured you will get weird behavior.
>
> QH
>
>
> On Mon, Oct 14, 2013 at 3:57 PM, Mike Bean <beandaemon at gmail.com> wrote:
>
> Well, I don't mean OU in the windows sense, I mean logical groupings of
> servers. Clusters. I'm trying to get my head around this, because
> intuitively, it just doesn't make any sense.
>
> If you're coming home from the grocery store, and you have more groceries
> then you can carry, you make multiple trips to the car.
>
> but I don't think it's quite that simple. Think networks aren't as smart
> as I think they are, intuitively, something configured to send as 1500,
> should be able to receive 1500 only, not necessary 9000, but something
> that's configured as 9000 should be able to send and receive as 9000 or
> 1500. (scratches head.)
>
> In actuality, B (9000) can send to A (1500)
> and A (1500) can send to B (9000)
>
> but if we set A to 9000, it can no longer send to B (9000)
> (scratches head)
>
> Boy, if you know someone who's in school looking for a secure job, tell
> them to become a network person. REALLY GOOD network people are worth
> their weight in gold.
>
>
> On Mon, Oct 14, 2013 at 3:26 PM, Quentin Hartman <qhartman at gmail.com>wrote:
>
> By OU, do you mean broadcast domain? ie - two LANs connected by a router?
> If so, it sounds like you have the (usually default) MTU of 1500 set on
> side A, but the 9000 set on B, and your router isn't correctly doing MTU
> management when crossing from one to the other.
>
> Going the other way works because the fragments are already small enough
> to be handled correctly at the other end. The complexity of making this
> work reliably is the primary reason so few people bother to use jumbo
> frames, even though they technically are better most of the time anymore.
>
> The wikipedia article on MTU is actually pretty good:
> http://en.wikipedia.org/wiki/Maximum_transmission_unit
>
> QH
>
>
> On Mon, Oct 14, 2013 at 3:15 PM, Mike Bean <beandaemon at gmail.com> wrote:
>
>
> So I've been scratching my head for a long time trying to understand
> this, and it makes no sense to me. We have to discrete OU's. SCP
> transmissions of any real size tend to fail from A to B, but not from B to
> A.
>
> My colleagues have been arguing that the fix is to set the MTU at side B
> to 1500. And Lo, and behold, when we do, it works, increase the MTU to
> 9000, and it fails again.
>
>
> http://unix.stackexchange.com/questions/14187/why-does-scp-hang-on-copying-files-larger-than-1405-bytes
>
> What I have difficulty understanding, is if a jumbo frame carries, say,
> 9000 bytes, and hypothetically, if it's a 5,000 bytes file + your MTU =
> 1500, which means you split it up into 3 transmissions.
>
> Same size transmission; capped packet size, more packets. So I would
> naturally conclude that at an MTU of 9000 it could get done in 1 what an
> MTU of 1500 would do in 3?
>
> (scratches head)
>
> _______________________________________________
> clue mailing list: clue at cluedenver.org
> For information, account preferences, or to unsubscribe see:
> http://cluedenver.org/mailman/listinfo/clue
>
>
>
> _______________________________________________
> clue mailing list: clue at cluedenver.org
> For information, account preferences, or to unsubscribe see:
> http://cluedenver.org/mailman/listinfo/clue
>
>
>
> _______________________________________________
> clue mailing list: clue at cluedenver.org
> For information, account preferences, or to unsubscribe see:
> http://cluedenver.org/mailman/listinfo/clue
>
>
>
> _______________________________________________
> clue mailing list: clue at cluedenver.org
> For information, account preferences, or to unsubscribe see:
> http://cluedenver.org/mailman/listinfo/clue
>
>
>
>
> _______________________________________________
> clue mailing list: clue at cluedenver.org
> For information, account preferences, or to unsubscribe see:http://cluedenver.org/mailman/listinfo/clue
>
>
>
> --
> +--------------------------------------+------------------------------+
> |Jim Bucks |Phone: +1(970)539-1242 Cell |
> |1924 44th Avenue Court | +1(970)330-3276 Home |
> |Greeley, Colorado 80634 (USA) |Email: jbucks at procci.com |
> +--------------------------------------+------------------------------+
> | WWW: http://www.procci.com |
> +---------------------------------------------------------------------+
>
>
> _______________________________________________
> clue mailing list: clue at cluedenver.org
> For information, account preferences, or to unsubscribe see:
> http://cluedenver.org/mailman/listinfo/clue
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://cluedenver.org/pipermail/clue/attachments/20131016/a9dd1926/attachment-0001.html
More information about the clue
mailing list