In recent weeks, Russian President Vladimir Putin announced a plan to enact measures to protect the Internet of Russia. In a speech to the Russian National Security Council he said, “we need to greatly improve the security of domestic communications networks and information resources.” Perhaps he should add Internet routing security to his list because, on a number of occasions in the past year, Russian Internet traffic (including domestic traffic) was re-routed out of the country due to routing errors by China Telecom. When international partners carry a country’s domestic traffic out of the country, only to ultimately return it, there are inevitable security and performance implications.
Last year, Russian mobile provider Vimpelcom and China Telecom signed a network sharing agreement and established a BGP peering relationship. However, as can often happen with these relationships, one party can leak the routes received from the other and effectively insert itself into the path of the other party’s Internet communications. This happened over a dozen times in the past year between these two providers. This is a general phenomenon that occurs with some regularity but isn’t often discussed in BGP security literature. In this blog post, we’ll explore the issue via the lens of this single example. In a follow-on blog, we’ll explore several additional examples.
How peering links can go wrong
In the world of BGP parlance, the term peering is overloaded. The same term can refer to a simple BGP adjacency between two autonomous systems (AS) as well as a particular type of relationship between two ASes where traffic is exchanged without money changing hands. This is also referred to as a “settlement-free” relationship and is the definition of peering used in this blog post.
Peering relationships between ISPs are quite common. By exchanging traffic directly without using a transit provider, each ISP saves on transit costs — although as transit prices fall the overhead of maintaining these peering relationships becomes less attractive. Regardless, from a routing standpoint, routes exchanged between two peering ISPs typically stay within the customer cone of each ISP. In the diagram below, ISP A sends routes from its customers to its peer ISP B. As a result, traffic from ISP B’s customers following those routes flows over the peering link to ISP A’s customers.
This arrangement can go wrong in a couple of different ways. ISP B could take the routes learned from ISP A and send them on to its providers and/or peers as illustrated in the graphic below. By doing this, ISP B effectively inserts itself into the path of traffic destined for ISP A from the outside world. We occasionally field questions from customers of Dyn IP Transit Intelligence product (formerly called Renesys Market Intelligence) who want to know why we reclassified one of their peers as a transit provider.
Typically after a little investigation, we determine that the peer is mistakenly passing their company’s routes on to the peer’s transit providers (or its peers) — effectively providing the company with free transit. In fact, one popular use case for IPTI is for ISP operators to monitor how their network’s adjacencies get classified. When there is a difference between what IPTI shows and how the operator thought his network was connected, someone isn’t propagating routes correctly.
The other way a peering relationship can go wrong is if ISP B announces routes learned from its providers and/or peers to ISP A. In this scenario (illustrated below), ISP A will then redirect traffic following these routes through ISP B, even if these routes are not normally handled by ISP B.
Unlike other routing leak scenarios, such as Indosat originating the entire global routing table or VolumeDrive leaking nearly the entire BGP table from one transit provider to another, the leaks described above occur with much greater frequency and with little fanfare. In fact, typically the parties involved are unaware of the glitch and, as a result, these more limited problems can persist much longer than the larger catastrophic incidents. Nonetheless, the results typically include traffic going where it wasn’t intended with a corresponding immediate hit to performance and the potential for security implications.
The Vimpelcom – China Telecom connection
 |  |
In September 2013, China Telecom and Vimpelcom (one of Russia’s “big three” mobile operators) announced an agreement to establish a direct network interconnection. We would have expected to see this relationship appear in BGP routing, either as a peering relationship or possibly a limited transit relationship to allow Vimpelcom to reach the Far East. However, on several occasions in the past year, we observed China Telecom mishandling routes it received from (and sent to) Vimpelcom, effectively inserting itself into the path of Internet traffic to and from Vimpelcom.
One of those occasions was the 5th of August of this year, when China Telecom (AS4134) announced routes from Vimpelcom (AS3216) to all of its peers for several hours (leak scenario 1 from above) which forced inbound traffic to Vimpelcom to go through China Telecom routers. During this incident, over 7,000 routes from Vimpelcom’s customer cone were globally announced by China Telecom with AS paths in the following form:
… {1299,3257,1273,3320,6461,174,…} 4134 3216 …
The following charts illustrate the impact of these incidents by looking at how traceroute measurements into Vimpelcom were affected. As the charts show, latencies spike as traffic is dragged out to China and through the molasses of China Telecom’s under-provisioned international links.
 |  |
 |  |
The following example traceroute illustrates an errant path taken by traffic during this routing leak. In this example, the traceroute begins in Reston, Virginia where NTT takes it to California and hands it off to China Telecom. China Telecom then takes it to Shanghai, China and then to its routers in Frankfurt, Germany before entering Russia en-route to Omsk.
trace from Reston, VA to Omsk, Russia at 12:00 Aug 05, 2014
1 *
2 129.250.204.153 (NTT America, Ashburn, US) 1.010ms
3 129.250.4.40 (NTT America, Ashburn, US) 0.934ms
4 *
5 129.250.5.13 (NTT America, San Jose, US) 74.649ms
6 129.250.8.6 xe-0.chinanet.snjsca04.us.bb.gin.ntt.net 70.909ms
7 202.97.90.33 (China Telecom, San Jose, US) 71.451ms
8 202.97.58.237 (China Telecom, Shanghai, CN) 341.819ms
9 202.97.58.66 (China Telecom, Frankfurt, DE) 641.424ms
10 118.85.204.58 (China Telecom, Frankfurt, DE) 608.965ms
11 79.104.245.250 pe-r.Omsk.gldn.net 632.054ms
12 195.239.162.178 (Vimpelcom, Omsk, RU) 687.536ms
13 89.179.76.25 vpn2-gi0-0.omsk.corbina.net 682.153ms
14 128.73.155.213 128-73-155-213.broadband.corbina.ru 707.525ms
As seen from within China, latencies to Vimpelcom via the direct link from China Telecom got lower and more stable, but did not improve dramatically. In fact, these traces were still getting directed to China Telecom’s routers in Frankfurt, Germany, and given the fact that overland round-trip latencies from Russia to China are around 120ms, the observed latencies suggest that such traffic may still go across submarine cables to reach Europe.
 |  |
If this routing arrangement is intended to provide Vimpelcom low-latency access to the Far East, it isn’t working that well. The examples below show latencies increasing during a routing leak from locations in the Far East.
 |  |
The August 5th event was one of the times that China Telecom briefly announced nearly a full BGP table (326,622 routes) to Vimpelcom, placing itself in the path of outbound traffic from Vimpelcom to the outside world — includingRussian routes. (Leak scenario 2 from above.)
Below is a traceroute illustrating the new path to a New Hampshire ISP. Vimpelcom takes the traffic to Frankfurt, hands it over to China Telecom which takes it to Shanghai before handing it over to Cogent in Los Angeles. Cogent takes it across the country to Fairpoint Communications in New Hampshire.
trace from Moscow to Manchester, NH at 12:09 Aug 05, 2014
1 *
2 194.154.89.125 (Vimpelcom, Moscow, RU) 0.743ms
3 79.104.235.66 mx01.Frankfurt.gldn.net 40.574ms
4 118.85.204.53 beeline-gw3.china-telecom.net 43.198ms
5 202.97.58.57 (China Telecom, Shanghai, CN) 302.433ms
6 202.97.58.238 (China Telecom, Los Angeles, US) 479.642ms
7 202.97.49.14 (China Telecom, Los Angeles, US) 487.225ms
8 38.104.139.77 te0-7-0-24.ccr21.sjc03.atlas.cogentco.com 380.087ms
9 154.54.6.105 be2000.ccr21.sjc01.atlas.cogentco.com 375.079ms
10 154.54.28.33 be2164.ccr21.sfo01.atlas.cogentco.com 371.727ms
11 154.54.30.54 be2132.ccr21.mci01.atlas.cogentco.com 372.585ms
12 154.54.6.86 be2156.ccr41.ord01.atlas.cogentco.com 370.596ms
13 154.54.44.86 be2351.ccr21.cle04.atlas.cogentco.com 367.498ms
14 154.54.25.89 be2009.ccr21.alb02.atlas.cogentco.com 371.972ms
15 38.104.52.78 (Cogent, Albany, US) 367.334ms
16 70.109.168.139 burl-lnk.ngn.east.myfairpoint.net 321.980ms
17 64.222.166.166 (Fairpoint Communications, Concord, US) 315.036ms
18 64.223.189.66 static.man.east.myfairpoint.net 321.682ms
But the path from Moscow to New Hampshire was always going to be long. Who’s to say that Los Angeles (or even Shanghai) might not be a reasonable waypoint under certain conditions? Alternatively, we can pick something closer to Moscow to illustrate how crazy this routing is. The traceroute below shows Vimpelcom taking traffic to Frankfurt, handing it over to China Telecom which takes it to Shanghai before handing it over to Chello Broadband, which peers with China Telecom in Los Angeles. Chello then takes it from New York to Frankfurt (again!) and then into the German countryside.
trace from Moscow, Russia to Marburg an der Lahn, Germany at 14:29 Aug 05, 2014
1 *
2 194.154.89.125 (Vimpelcom, Moscow, RU) 0.908ms
3 79.104.235.74 mx01.Frankfurt.gldn.net 40.695ms
4 118.85.204.49 beeline-gw2.china-telecom.net 48.799ms
5 202.97.58.61 (China Telecom, Shanghai, CN) 290.810ms
6 202.97.58.202 (China Telecom, Los Angeles, US) 514.115ms
7 202.97.90.62 (China Telecom, Los Angeles, US) 537.933ms
8 213.46.190.217 213-46-190-217.aorta.net (New York) 414.365ms
9 84.116.135.122 (UPC Austria GmbH, Vienna, AT) 420.591ms
10 213.46.160.205 uk-lon02a-rd1-pos-4-0-0.aorta.net 421.530ms
11 84.116.133.65 (UPC Austria GmbH, Frankfurt, DE) 421.557ms
12 81.210.129.234 7111a-mx960-01-ae1.fra.unity-media.net 420.867ms
13 80.69.107.214 7111a-mx960-02-ae0.fra.unity-media.net 418.427ms
14 80.69.107.185 7211a-mx960-01-ae1.gie.unity-media.net 420.454ms
15 88.152.128.0 (Unitymedia, Marburg an der Lahn, DE) 423.105ms
Even Internet paths from Moscow to other parts of Russia were forced through China Telecom’s routers. In the following example, a traceroute from Moscow is taken by Vimpelcom to Frankfurt, handed over to China Telecom’s routers in Frankfurt and, (mercifully) redirected back into Russia via Megafon without getting directed out to Shanghai. (This diversion of domestic Russian traffic is illustrated in the graphic at the beginning of this blog.)
trace from Moscow, Russia to Yaroslavl, Russia at 13:13 Aug 05, 2014
1 *
2 194.154.89.125 (Vimpelcom, Moscow, RU) 0.542ms
3 79.104.235.74 mx01.Frankfurt.gldn.net 37.006ms
4 118.85.204.57 beeline-gw4.china-telecom.net 39.505ms
5 213.248.84.185 ffm-b10-link.telia.net 41.481ms
6 62.115.137.180 ffm-bb2-link.telia.net 42.227ms
7 80.91.251.159 s-bb4-link.telia.net 42.894ms
8 213.155.133.105 s-b2-link.telia.net 41.528ms
9 80.239.128.234 megafon-ic-151430-s-b2.c.telia.net 42.707ms
10 *
11 78.25.73.42 (MegaFon, Volga,RU) 49.992ms
12 213.187.127.98 ysu1-ccr1036-1.yar.ru 50.301ms
13 213.187.117.230 (NETIS Telecom, Yaroslavl’, RU) 54.769ms
Conclusion
In September, ACM Magazine published a terrific piece by Boston University Professor Sharon Goldberg which asked “Why is it taking so long to secure Internet routing?”. She does a great job covering the issues surrounding routing security and the efforts thus far to address its vulnerabilities. I would add peering leaks like those described in this blog to her list of unresolved issues, problems that are much less well known or even discussed. At present, the best defense is to monitor how your routes traverse the Internet (such as with Dyn Internet Intelligence) and to carefully filter the routes you receive. Without both measures, your traffic could be easily misdirected, potentially hurting both the performance and security of your Internet communications.
Investors also weren’t too excited about Putin’s Internet security plan, causing Vimpelcom’s stock to drop to a record low, but maybe they were disappointed that it didn’t address routing security.
No comments:
Post a Comment