Tutorial 9: Internet Routing in Practice Expanded

Learning Objectives

• Describe the Internet's hierarchical ISP structure (Tier-1, Tier-2, Tier-3) and their roles.
• Explain the concepts of peering, transit, and settlement-free interconnect.
• Analyze the role and operation of Internet Exchange Points (IXPs) in peering.
• Evaluate multi-homing strategies and their impact on routing and fault tolerance.
• Understand route aggregation, de-aggregation, and their implications for routing table growth.
• Examine real-world routing policies and how BGP attributes enforce them.
• Describe the function of route servers at IXPs.
• Apply BGP communities and attributes for traffic engineering.
• Analyze case studies of ISP and cloud provider routing architectures.
• Identify current challenges in Internet routing: table growth, convergence, and security.

Overview

While the previous tutorials covered the theoretical aspects of BGP and hierarchical routing, this tutorial focuses on the practical realities of Internet routing. The Internet is a complex ecosystem of thousands of Autonomous Systems (ASes) with diverse business relationships: customer-provider, peering, and transit. These relationships determine how routes are propagated and how traffic flows. Internet Exchange Points (IXPs) are critical for efficient peering. Multi-homing, route aggregation, and policy-based traffic engineering are everyday practices for network operators. This tutorial explores these topics in depth, using real-world examples and case studies to illustrate how the Internet's routing system operates in practice. We will also examine current challenges such as the growing size of the global routing table, convergence delays, and security issues.

Detailed Technical and Theoretical Content

1. The Internet's Hierarchical ISP Structure

The Internet's AS-level topology is often described as a hierarchy:

• Tier-1 ISPs: These are global backbone providers that have no upstream providers (they do not purchase transit). They peer with each other to exchange traffic. Examples: AT&T, Level 3, NTT, Deutsche Telekom. They form a dense peering mesh.
• Tier-2 ISPs: These are regional or national providers that purchase transit from Tier-1 ISPs and may also peer with other Tier-2 ISPs. They may also have their own customers (Tier-3 or enterprises).
• Tier-3 ISPs: These are local access providers that purchase transit from Tier-2 or Tier-1 ISPs. They typically serve end-users and small businesses.

This hierarchy allows the Internet to scale; most routing information is aggregated and propagated upward.

2. Peering, Transit, and Settlement-Free Exchange

AS relationships are governed by economic agreements:

• Transit: An AS pays another AS to carry its traffic to the rest of the Internet. The provider announces all routes to the customer and forwards traffic for the customer.
• Peering: Two ASes agree to exchange traffic between their customers (and themselves) without payment (settlement-free). They do not carry transit traffic for third parties. Peering reduces costs and improves performance.
• Customer-provider: The customer pays the provider; the provider announces the customer's routes.

These relationships are enforced by BGP policies: routes from a customer are usually advertised to all peers and providers; routes from a peer are only advertised to customers (not to other peers or providers) to avoid providing free transit.

3. Internet Exchange Points (IXPs) – Architecture and Operation

An IXP is a physical infrastructure (often a Layer 2 switch fabric) where multiple ASes meet to exchange traffic. At an IXP, each participant has a router connected to the exchange fabric, typically via a VLAN. They can establish BGP sessions (eBGP) with each other. IXPs reduce the need for transit, improving latency and cost. Popular IXPs include AMS-IX, DE-CIX, LINX. Many IXPs use route servers to facilitate peering: participants establish BGP sessions with the route server, which then reflects routes to other participants, reducing the number of BGP sessions required.

4. Multi-homing Strategies and BGP Load Balancing

An AS may connect to multiple upstream providers (multi-homing) for redundancy and performance. Multi-homing requires the AS to have its own ASN and PI (Provider Independent) address space (or use provider-aggregatable address space with careful configuration). With BGP, the AS can:

• Outbound load balancing: Use LOCAL-PREF to prefer one provider for certain destinations, or use BGP attributes to influence path selection.
• Inbound load balancing: Use AS-PATH prepending, MED, or communities to influence which provider is used to enter the AS.
• Failover: If one provider fails, BGP withdraws routes, and traffic automatically shifts to the other.

Multi-homing also introduces challenges: the AS must ensure that its announced prefixes are reachable from both providers, and it must handle asymmetric routing.

5. Route Aggregation and De-aggregation in Practice

Route aggregation (CIDR) is essential to keep the global BGP table manageable. However, operators often de-aggregate prefixes (announce more specific routes) for traffic engineering purposes, to attract traffic to specific entry points or to load balance. This practice contributes to the growth of the BGP table. The trade-off between aggregation and fine-grained control is a constant challenge.

6. Real-World Routing Policies and Their Implementation

Operators use BGP policies to enforce business agreements and optimize performance. Common policies include:

• Prefer customer routes over peer routes, and peer routes over provider routes: This is the typical "valley-free" routing.
• Manipulate LOCAL-PREF: Set higher LOCAL-PREF for routes from preferred providers or for specific prefixes.
• Filtering: Ingress filtering to reject invalid prefixes (e.g., using IRR or RPKI).
• Route tagging with communities: Use communities to signal to other ASes (e.g., do not advertise to peers, or set specific LOCAL-PREF).

7. Route Servers at IXPs

At an IXP, a route server is a BGP speaker that does not forward data but facilitates peering. It allows participants to exchange routes without having to establish a separate BGP session with every other participant. The route server operates under policies set by the IXP, often limiting announcements to certain prefixes. This greatly reduces the number of BGP sessions needed at large IXPs.

8. Traffic Engineering with BGP Communities and Attributes

BGP communities are widely used for traffic engineering. Examples:

• Prepend community: Tell a provider to prepend its AS to routes (to make them less preferred).
• MED community: Set MED values to influence inbound path selection.
• Filtering communities: Request that a provider does not advertise a route to certain peers.

Many ISPs publish their community policies so customers can influence routing.

9. Case Studies: Tier-1, Tier-2, and Cloud Provider Routing

• Tier-1 ISP: They maintain a full BGP table and have peering relationships with most other Tier-1s. They use BGP to manage traffic across their global backbone, often using MPLS in their core.
• Tier-2 ISP: They buy transit from Tier-1s and peer at IXPs. They may use BGP communities to influence how their upstream providers handle their routes.
• Cloud Provider (e.g., AWS, Azure): They have multiple data centers and BGP routing to announce their IP prefixes. They use BGP for traffic engineering, e.g., to direct users to the closest data center (using GEO-based routing with BGP anycast). They also use BGP communities to signal routing preferences.

10. Challenges: Routing Table Growth, Convergence, and Security

Current challenges in Internet routing:

• Table growth: The global BGP table exceeds 1 million prefixes. This strains router memory and CPU.
• Convergence: BGP convergence is slow (minutes) because of hold-down timers and the need to propagate updates across multiple ASes. This causes temporary black holes during failures.
• Security: Prefix hijacking and route leaks remain issues despite RPKI deployment.
• De-aggregation: Operators de-aggregate for traffic engineering, which worsens table growth.

    Customer announces routes to Provider
    Provider announces routes to all other providers and peers
    Provider does NOT announce peer routes to other peers (valley-free)
    

Quiz

Answer each question; check your understanding by revealing the answer.

Question 1: What is a Tier-1 ISP?

Question 2: What is the difference between peering and transit?

Question 3: What is an Internet Exchange Point (IXP)?

Question 4: What is multi-homing?

Question 5: What is route aggregation (CIDR) and why is it important?

Question 6: What is AS-PATH prepending used for?

Question 7: In the ISP hierarchy, which type of ISP typically has no upstream providers?

Question 8: What is a route server at an IXP?

Question 9: How does an AS typically treat routes learned from a customer versus a peer?

Question 10: What is the valley-free routing principle?

Question 11: What is the primary benefit of peering at an IXP?

Question 12: How can an AS influence inbound traffic?

Question 13: What is the main cause of BGP table growth?

Question 14: How does route aggregation help with routing table size?

Question 15: What is a community in BGP?

Question 16: Why might an AS de-aggregate its prefixes?

Question 17: What is the typical BGP convergence time on the Internet?

Question 18: What is a common policy for a Tier-2 ISP regarding its upstream providers?

Question 19: What is the role of the IRR (Internet Routing Registry) in routing?

Question 20: What is BGP route flap damping and why is it used?

Question 21: What is the relationship between a Tier-1 ISP and a Tier-2 ISP?

Question 22: How does an IXP route server differ from a traditional router?

Question 23: Why is RPKI important in practice?

Question 24: What is the impact of de-aggregation on the BGP table?

Question 25: How do cloud providers use BGP for global load balancing?

Exercises

Work through these problems; sample solutions are hidden.

Exercise 1: Draw a simple topology showing a Tier-1 ISP, two Tier-2 ISPs, and three Tier-3 ISPs. Indicate peering and transit relationships with arrows.

Exercise 2: An enterprise AS has two upstream providers: ISP-A and ISP-B. It wants to use ISP-A as primary for outbound traffic. How can it achieve this using BGP attributes?

Exercise 3: Explain why a route server at an IXP does not need to forward data packets.

Exercise 4: An AS wants to attract inbound traffic from ISP-A but not from ISP-B. How can it achieve this using AS-PATH prepending?

Exercise 5: What is the effect of route aggregation on the routing table of an upstream provider? Provide an example.

Exercise 6: Compare the cost and performance trade-offs between using transit and peering.

Exercise 7: What is the role of BGP communities in a multi-homed AS? Provide a specific example of a community used to request prepending.

Exercise 8: Explain how an IXP can reduce the number of BGP sessions needed for a participant with 100 peers.

Exercise 9: What is the valley-free routing violation and what can happen if an AS violates it?

Exercise 10: A cloud provider wants to announce the same IP prefix (e.g., 198.51.100.0/24) from two data centers: one in New York and one in San Francisco. How can they use BGP to steer users to the closest data center?

Exercise 11: How does route flap damping affect convergence? What are the downsides?

Exercise 12: Compare the routing policies of a Tier-1 ISP versus a small enterprise AS.

Homework Assignments

These questions require deeper thought and research. Write comprehensive answers.

Homework 1: Research the peering policies of a major Tier-1 ISP (e.g., AT&T, NTT). Explain what prefixes they announce, how they decide to peer, and what their BGP community offerings are.

Homework 2: Explain in detail the process of route propagation from a stub AS to the global BGP table. How does aggregation at each level reduce the number of prefixes?

Homework 3: Compare and contrast the use of route servers at IXPs with the use of route reflectors in iBGP. What are the similarities and differences?

Homework 4: Analyze the impact of de-aggregation on the growth of the global BGP table. Provide statistics and discuss the trade-offs between traffic engineering and table size.

Homework 5: Design a BGP policy for a multi-homed enterprise that wants to use two providers (P1 and P2) and also peer with a content delivery network (CDN) at an IXP. Specify the attributes and filtering rules for outbound and inbound traffic.

Homework 6: Explain the concept of "BGP anycast" and provide two use cases (e.g., DNS, CDN). How does it work in terms of routing?

Homework 7: Discuss the security challenges in Internet routing and how RPKI and BGPsec address them. What are the limitations of these solutions?

Homework 8: Investigate the role of the IRR (Internet Routing Registry) and how operators use it for route filtering. Provide an example of an IRR entry.

Homework 9: Compare the routing architectures of a traditional Tier-2 ISP with a large cloud provider (e.g., AWS). What are the main differences in scale, policy, and traffic engineering?

Homework 10: Explain the concept of "route reflector clients" in iBGP and how they differ from non-clients. How does a route reflector handle routes from clients vs non-clients?

Homework 11: Analyze the BGP convergence process after a link failure between two Tier-1 ISPs. What factors affect the convergence time and how can it be improved?

Homework 12: Research the concept of "BGP communities" and how they are used for traffic engineering in practice. Provide at least three examples of well-known communities used by major ISPs.

Summary

This tutorial provided a practical view of Internet routing, covering the ISP hierarchy, peering and transit, IXPs, multi-homing, route aggregation, and real-world policies. We examined how BGP attributes and communities are used for traffic engineering, and explored case studies of different network types. We also discussed current challenges such as routing table growth, convergence, and security. Understanding these practical aspects is essential for network operators and engineers who manage Internet-connected networks. The next tutorial will delve into Software-Defined Networking and OpenFlow, which represent a paradigm shift in control-plane architecture.

End of Tutorial 9 .