Understanding the architecture of the Internet

The Internet consists of a plethora of networks (public, private, government etc.) which carry an abundant range of information, services and resources and connect to each other via bundles of data called packets. Everything that we use in our daily lives such as E-mail, Search Engines, VoIP, Peer-to-Peer file sharing are a result of the transference of these data packets. Shipping packets around the world involves cooperation between different organizations and can be quite challenging. Packets can also be lost during transmission due to the distance, collision or excessive crowding. This packet transfer across the Internet is made easier by Internet eXchange Points (IXP) where local networks efficiently exchange information at a common point within a country. At an IXP, Internet Service Providers (ISPs) exchange traffic using high speed network/Ethernet switch. An Internet Service Provider is a company or organization that provides web access to its clients by maintaining a direct line to the internet and web servers. AT&T, Hathway, Verizon, Bharti Airtel, Sify are some well-known Internet Service Providers.

Image Showing ISP Scenario With and Without IXP

Fig. 1: Image Showing ISP Scenario With and Without IXP

The above figure depicts the ISP scenario both without an IXP and with an IXP. In case of an IXP, every ISP has to buy just one whole circuit from their ISP to the IXP instead of several half circuits to every other ISP. This removes the complexity of direct one-to-one connections between ISPs as well as reducing the cost for further connections. Through the IXP, traffic is exchanged locally within the country rather than using the local Internet traffic overseas. This saves international bandwidth which can be used for backup or international traffic. A comprehensive layout of the Internet hierarchy is shown below.

Image Showing Comprehensive Heirarchy of Internet

Fig. 2: Image Showing Comprehensive Heirarchy of Internet

IXP Architecture

There are usually two kinds of IXP architectures: Layer2 and Layer3. In Layer 2 (L2), all the ISPs are interconnected using Ethernet switch whereas in Layer 3 (L3), all the ISPs are interconnected using a router. Layer 3 is not widely used as it is router based and thus was overwhelmed by the rapid growth of the Internet. Today, it is merely a marketing concept used by Transit ISPs. Layer 2 topology uses a common network medium like Ethernet (100 Mbps/1 Gbps/10 Gbps/100 Gbps technologies) which is connected to neighbouring ISPs. Members bring their own routers and circuits from their backbone and they are free to set up peering agreements with each other as they wish. All traffic is exchanged outside routers that are connected to a shared media (Ethernet switch).

Diagram showing IXP Architecture

Fig. 3: Diagram Showing IXP Architecture

A Layer 2 IXP has a proven upgrade path with a simplistic design. Also, it can scale up to 60 IXP members with low cost entry for its participants. Services are offered such as Search engines, Internet portal, DNS Root, Looking Glass etc. ISPs also use two switches for load-sharing or redundancy. In a Layer 2 exchange, each member has to peer with every other member as part of their IXP membership. This is called Mandatory Multi-lateral Peering.

Peering at an IXP using BGP

An ISP can exchange traffic with any of the other ISPs through the IXP switch (public peering) or directly by having a physical interconnection (private peering). Peering between ISPs requires exchange of routes used between Autonomous Systems (AS). Autonomous Systems (AS) are independently run networks or a set of related networks, some being commercial ISPs which come under one administrative domain. For successful peering, the ISP router needs to be able to run BGP, an Ethernet port to connect to the IXP switch and a WAN port to connect to the ISP backbone’s WAN media. The peering is done through the Border Gateway Protocol (BGP) which is basically a path vector routing protocol that exchanges routes for IP address ranges or prefixes.

Image showing Peering at IXP with BGP

Fig. 4: Image Showing Peering at IXP with BGP

The BGP routes between domains and unifies network organizations while helping in choosing the best route by using preferences local to AS and AS path length. As a BGP route travels from AS to AS, the AS number of each AS is stamped on it when it leaves that AS which is called the AS-path. IXPs themselves do not require Autonomous System Numbers (ASN) as the Ethernet switch does not run BGP, but the services run by the IXP requires a transit router that needs a Public ASN and Address space as well. ASNs make it easier for the BGP to route through IXP traffic. Each participating ISP in an IXP needs to run BGP for peering to take place as external BGP configures directly with other participants in the IXP. The BGP operation is shown below.

Block Diagram Showing BGP Operation

Fig. 5: Block Diagram Showing BGP Operation

Salient features of an IXP:

· Neutral, accessible and secure.

· Reliable and Redundant due to multiple switches.

· L2 IXP is the most common architecture used where the core is the Ethernet switch.

· IXP traffic steadily increases with increase in member Autonomous Systems.

· Majority of the IXP traffic comes from Tier-2 ISPs.

· Most of the Peer-to-Peer traffic at IXP remains invisible.

· BGP based efforts for discovering Peer-to-Peer links in AS levels have limited success and provides at best lower bound for number of links in the internet. In contrast, number of Peer-to-Peer links at a single IXP exceeds even lower bounds.

· IXP has a diverse ecosystem in terms of member ASes, peering strategy, traffic and geographical coverage that mimics the Internet’s AS ecosystem.

Case Study of NIXI

Case Study of NIXI

We looked at some of the fundamentals of the Internet’s network hierarchy namely, Internet eXchange Point (IXP), Internet Service Provider (ISP), Border Gateway Protocol (BGP) and Peering. This article aims to provide a practical viewpoint of the above concepts along with its applications. To get a good understanding of the ISP network, let us consider a local IXP – National Internet eXchange of India (NIXI). It is a neutral exchange point of ISPs setup in 2003 and funded by the Government of India. NIXI was set up for peering of ISPs among themselves for the purpose of routing the domestic traffic within the country, instead of taking it all the way to US/Abroad, thereby resulting in reduced latency and saving on International Bandwidth. Currently, it has 7 exchange points in Delhi (Noida), Mumbai, Hyderabad, Bangalore, Chennai, Kolkata and Ahmedabad.

Fig. 6: Image Showing Home Screen of NIXI

NIXI Tool: Looking Glass

We can use the NIXI website for gathering data and performing an analysis of ISPs with various tools, one of which is a Looking glass. A Looking glass, as it is aptly named, provides public view of routing information available at an IXP by making route collector routes available for global view. It also provides a web interface for running a limited set of commands on routers. The route collector makes use of a router/UNIX system running Border Gateway Protocol (BGP) to gather routing information from service provider routers at an IXP by peering with each ISP. It does not forward packets, only holds the IXP routes. NIXI uses Layer 2 IXP architecture where all the ISPs are interconnected with an Ethernet switch. NIXI also implements the multilateral mandatory peering policy among all its participants.

When using the NIXI looking glass, we use the ‘show ip bgp summary’ command to list out the routing and connection data of the ISPs. The command produces a lot of parameters to take into consideration. The typical output that we obtain from the NIXI site is as shown below:

Fig. 7: Obtaining Output a Standard Output from NIXI Site

The individual parameters of the command output are as follows:

BGP router identifier – It is the IP address (218.100.48.1) that all other BGP speakers recognize as representing the BGP router. It can be defined as the highest IP address or the router identifier specified by the bgp router-id command.

Local AS number – Autonomous System with which the BGP router is associated (24029).

BGP Table Version – Internal version number of BGP database (508508).This number increases in increments when BGP table changes.

Main routing table version – It is the last version of BGP database that was injected into the main routing table (508508).

Network entries – number of unique prefix entries in the BGP database (13163 using 1540071 bytes of memory).

Path entries – Number of path entries in the BGP database (14844 using 771888 bytes of memory). Only a single path entry will be installed for a given destination. If multipath routes are configured, a path entry will be installed for each multipath route.

BGP path/best path attribute entries – Number of unique BGP attribute combinations for which a path is selected as best path (1606/1372 using 224840 bytes of memory).

BGP AS-PATH entries – Number of unique AS-PATH entries (1116 using 29944 bytes of memory).

BGP community entries – Number of unique BGP community attribute combinations (275 using 11024 bytes of memory).

BGP route-map cache entries – Number of BGP route-map match and set clause combinations. Here, 0 indicates that the route cache is empty.

BGP filter-list cache entries – Number of filter-list entries that match an AS-path access list permit or deny statements. Here, 0 indicates that the filter-list cache is empty.

Total bytes of memory – Total amount of memory, in bytes, used by the BGP process.

BGP activity – Displays the number of times that memory has been allocated or released for a path or prefix.

Fig. 8: Command Output Table

The command output also contains a table towards the end; the columns of the table are further elaborated below:

Neighbour – IP address of the neighbour with which this router is setting up a relationship. This is available in the Connected Networks section of the website.

V – The version of BGP this router is running with the listed neighbour.

AS – The listed neighbour’s Autonomous System number.

MsgRcvd – The number of BGP messages received from this neighbour.

MsgSent – The number of BGP messages sent to this neighbour.

TblVer – It is the last version of the BGP database table that was sent to this neighbour. It is basically the router table version. Under normal circumstances, this should be the same if routing is stable.

InQ – The number of messages from this neighbour that are waiting to be processed.

OutQ – The number of messages queued and waiting to be sent to this neighbour. TCP flow control prevents this router from overwhelming a neighbour with a large update.

Up/Down – The length of time this neighbour has been in the current BGP state (established, active, or idle). If the BGP session to the neighbour is up in the “Established” state, this is the duration of the current TCP session. If it’s not up, this will display the current state of the connection.

State/PfxRcd – Current state of BGP session: active, idle, open sent, open confirm, or idle (admin). If the state field shows active, the router is actively attempting to create a TCP connection to this neighbour. If the session is in the established state, a state is not displayed; instead, a number representing the PfxRcd is displayed. If the TCP session is up and the routers are exchanging routes, then this field will be a numeric value that shows how many route prefixes (BGP network entries) have been received from the remote neighbour or peer group.

NIXI Tool: MRTG Statistics

We can also use MRTG statistics to find traffic graphs for different locations.

Fig. 9: Graph Showing Aggregated Traffic For All Locations in NIXI Tool: MRTG Statistics

Applications

The NIXI site gives us the Autonomous System Number (ASN) by which we can identify the network. This information from the NIXI website, coupled with tools like Looking glass can be put to use in several applications, some of them which include:

· Creating a graphical representation of the ISP network showcasing the interconnection between various ISPs.

· Finding the Hop count between ISPs and thus calculating the cost incurred to them to send or receive their packets through a particular ISP.

· Analysis of shortest path, One-hop and Multi-hop routing between two ISPs.

· Identification between routing pattern of different ISPs which can be helpful in further navigation.

· Several in-depth analyses like Traffic vs. increasing Bandwidth vs. Access which can be worked upon to derive certain case studies.