[clue-tech] multihomed ISP and BGP

[Nate Duehr]

Jim Ockers wrote:

> "$ISP says that they have no preferred provider.  They say they are running 
> BGP, but they don't use weighted preferential routing system, and have no 
> plans to introduce one."
> 
> Is this the correct approach?  My concern is that if $ISP advertises their
> IP addresses via BGP on backbones A B and C, how would a different network
> (who is also peered with A B and C) choose which backbone to transit the
> traffic through?

That's kinda all out of your hands once it leaves the local BGP router. 
  The nature of BGP, really.

Answering whether it's the correct approach is kinda like asking if you 
should vote Republican or Democrat... they both suck.  It's all about 
long-term trends.

Having lived through this at an ISP -- the original setup was 
non-weighted and we had backbone connections to AT&T, Verio, and UUNet 
back in the day.

One carrier had pre-pended their own AS at multiple hops, another 
carrier had their entire network looking like one AS to us, and the 
other had a mixture of all of the AS's of the companies they'd bought 
and acquired over the years.

You could control inbound -- kinda -- by prepending your own AS multiple 
times on the router you did NOT want to route back INTO yourself 
through... so if you got complaints that a particular carrier was having 
packet loss or other problems, you could make yourself look "further 
away" to them to try to get traffic to route in through the other carriers.

It was an utter mess.  The only way to "fix" BGP is to actually TEST the 
routes, not something easily done... and even then, you only had 
"control "of outbound.

The company was working on some really smart code to do just that... it 
would look at the BGP tables in our edge routers, and forcibly route 
between our 18 sites on whatever paths looked available, not just the 
best path, measure speed/latency/loss, and then dork with pre-pending 
our AS in a "live" way into our edge-routers to force traffic to the 
"technically best right now" routes from whatever carrier that was.  The 
problem was, the carriers also charged different rates for overages, 
etc... so business rules also had to be programmed into such a thing.

It never really got off the ground.  They even looked into patenting it, 
to be honest.

BGP generally is utterly broken today.  It was great for multi-homing 
back when multi-homing was "hard", but the next step needs to be a 
protocol everyone agrees on that actually measures PERFORMANCE across 
the available announced routes.

(And of course, there's always the loveliness of someone announcing 
something they don't own -- early BGP allowed too much of that -- 
nowadays good carriers only let people announce their own AS # and throw 
out any other announcements.)

> In particular I don't think I like asymmetric links, where the traffic goes
> one way them->$ISP->A->$OURISP->us but the other way is us->$OURISP->B->$ISP->them.
> Especially if B is a crappy network, which it might or might not be, I
> have no way of knowing.  If the traffic came & went over backbone A then at
> least we could get consistent round trip transit times, maybe.

Yep, you've got it.  Like I said, you can pre-pend your own AS a bunch 
of times (and even then, this only works if your upstream does NOT 
AGGREGATE your AS announcements!  That was a battle we fought for a long 
time to gain "control" over our multi-homed envrionment back then!) to 
try to force the traffic to go where you want it to.

In your case, you have ZERO control over any of this, since it's the 
guys upstream of you.  That would be tough.  You kinda just have to quiz 
them to see if they really pay any attention to degraded service and 
will take the appropriate steps needed to either "trick" the traffic to 
another backbone link, or in extreme cases -- DOWN the interface to the 
bad backbone until things get better.

If they're a "hands-off" shop and playing the "let's hope BGP always 
does the right thing"... run away and find a better hosting environment.

Ask hard questions, see what they say.  Don't like the answers, take 
your business elsewhere, I'd say -- but -- realize that having humans 
sitting there making routing decisions (because BGP sucks -- and yes, it 
does...) isn't cheap.   You kinda get what you pay for... and the bosses 
paying for it will NOT understand the difference.

> Questions for the group:
> 
> 1. Am I justified in preferring symmetric links over asymmetric links?  Why or
> why not?

Generally no.  Because a multi-homed environment instantly takes all of 
that out of the control of even your upstream.  The "idea" is that all 
backbones are created equal, but -- as you can see above -- they're not. 
  We were constantly downing the Verio link back then, because whenever 
it was up, people complained.  AT&T and UUNet generally did a good job 
of maintaining their performance, but we had individual days (throughout 
many years) where one or the other was suffering from a fiber cut 
somewhere, and their backbone was hosed... and during those events, 
nothing worked better than staying on the NANOG and other lists, seeing 
the reports of where the outages were, and manually downing interfaces 
that pointed into the "messy corners" of the Net.)

> 2. Is $ISP justified in not having any weighted preferential networking on
> any of their connections?  Why or why not?


They're saving money on staff.  That's about it.  Either they truly 
thing BGP always does a good job, or they're hoping and wishing.  None 
of those will work, long-term.


> 3. Is there any way to force symmetric links between two IP addresses that
> are connected via multiple backbones?  Or is this just not possible?


Not really.  Everyone is very leery to put in static routes, and rightly 
so... the Internet is supposed to "route around" problems, and no one 
really wants to break that by forcing things to go somewhere all the 
time.  But there are "tricks" that can be done to gently nudge the 
routing where you want it, and there's always the "nuclear" option of 
nuking an interface -- or maybe a little nicer -- stop making BGP 
announcements out that interface altogether.  Play dead and see if the 
packets find their way to the other router.


> I expect Nate might have an idea but I know there are lots of smart people
> on this list including a ton of lurkers.  So, thanks a lot in advance for
> the advice.

Ha... that's my thoughts.  I'll disclose that I was NOT one of the 
"router guys" -- we had CCIE's for this junk and one very smart 
programmers working on the "solution" that never really took hold. 
Maybe the company got some of that "stuff" done later, I don't know. 
The programmer was doing all sorts of work on it (including hacking 
inside the guts of the open-source BGP implementations on Linux, so he 
could use Linux as a "fake edge router" in our environment to gather up 
the possible routes across the "Net").  Other struggles included mainly 
aggregation problems... one backbone aggregated everything as if to say, 
"Just send it all to us, we'll take care of it.  Nothing to see here, 
move along!"


> PS I miss having InterNAP service.  Wasn't my decision to terminate it.

InterNAP made a business out of doing all this hard "manual" stuff, and 
they have some patents and similar types of thinkers that do things like 
actually CHECKING the routes available to them at any point in time. 
Their stuff ALWAYS worked when we had InterNAP plugged into a data 
center.  Seeing performance issues or outages from them was ultra-rare, 
and they were in a serious world of hurt somehow (fiber "back-hoe fade") 
if they were causing us problems -- they have humans watching things. 
Not cheap at all, either.  Almost too expensive... since we were already 
multi-homed, having their pipe was overkill -- but it did always work.

There's also things going on upstream that hurt the end-users that the 
providers do that are amazing... things like peers refusing to route for 
each other, and things like that.  The "best" (bad word to use, it's 
BAD) examples of this are the battles that have been raging on again, 
off again, for years between "traditional" backbones and Cogent.

Cogent keeps their rates low by transiting everything across someone 
else's backbone, and their original agreements were written in such a 
way as to make the other carriers think that it was a two-way street. 
But Cogent then under-cut everyone on price (and still does in most 
cases) and never built their own backbone, so the other carriers found 
their "peering deals" with Cogent really sucked... they were upgrading 
their infrastructure to deal with Cogent's traffic, and paying for their 
half of the transport infrastructure, but Cogent was getting all of the 
customers.

Level 3 has cut off peering without PUBLIC warning (I'm pretty sure they 
warned Cogent, but neither side is talking) with Cogent multiple times, 
for example.

All this "variability of service" for packet-switched networks is silly, 
but ultimately WAY back in the day, it's all because of deregulation. 
"You get what you pay for" has never been more true.   Everyone wanted 
open markets, and packet-switched networks across multiple carriers... 
and they got it... now they complain that it's hard to determine if 
you're really getting what you pay for.  Uh-huh.  Not a surprise 
considering the technology (BGP) under the hood.

You almost (as a end-user or small non-backbone ISP, or whatever) have 
to dedicate a certain percentage of your paid-for bandwidth for constant 
performance testing... and if you're not a multi-site sized customer... 
where do you test to?

I haven't seen any co-ops of small companies that want to run continuous 
geographically-distributed testing pop up yet, and I was almost CERTAIN 
we'd see that happen back in the late 90's.  It's almost a decade later, 
and there's still no real way to test the performance of your upstream 
effectively... what'cha gonna do?  Ping Yahoo?  Heh.  No one on the Net 
really knows if they're getting what they pay for.  Are there times of 
day our backbone connectivity slows because a router is over-subscribed? 
  Are there packet loss issues that always happen through a particular 
router in the path?  Are there latency issues on certain routes?

It's just kinda "don't ask don't tell" for this stuff, which seems 
crazy, since it's possible to build real tests and metrics.  The 
carriers just say, "Don't worry about it -- we'll do those tests."  But 
you can't prove/disprove that they do.  All you can do is call in 
trouble-tickets when their routes stink (performance-wise) for some reason.

BGP is great about routing around routes that are ALL THE WAY DOWN, but 
isn't the right answer for routing around performance bottlenecks. 
That's what I think, anyway...

Nate WY0X