A. Greenberg, J. R. Hamilton, N. Jain, S. Kandula, C. Kim, P. Lahiri, D. A. Maltz, P. Patel, S. Sengupta, “VL2: A Scalable and Flexible Data Center Network,” ACM SIGCOMM, (August 2009). [PDF]

Summary

VL2 is a scalable data center architecture to provide uniform high capacity between servers, performance isolation between services, Ethernet layer 2 semantics in large data centers, and dynamic resource allocation across large server pools (aka agility). The authors pose agility as the main motivation behind their design citing that without agility the server pool will be fragmented, and there will be hot-spots, congestion, lack of isolation between services, and large management overhead. VL2’s design promotes changes in the end hosts and server protocol stacks to work around existing switching hardware limitations (by leaving them unmodified). VL2 uses flat addressing, Valiant Load Balancing (VLB), end host based address resolution, Clos network topology, and make use of the existing protocols and technologies.There is a separation of identity and location of end hosts, and VL2 uses a directory service to keep track of the mappings.

One of the major contributions of this paper is the detailed traffic characteristics analysis of a large data center. The authors observed that

• majority of flows are small,
• more than 50% of the time each machine has about 10 concurrent flows, but at least 5% of the time it has greater than 80 concurrent flows,
• traffic pattern is divergent and cannot easily be summarized,
• there is no structure in traffic pattern,
• most failures are small in size,
• in 0.3% of failures all redundant (1:1 redundancy) components in a network device group become unavailable.

The design principles and the actual design of VL2 are directly influenced by these observations:

• VL2 replaces conventional hierarchical topologies with Clos network topology that ensures graceful degradation of bandwidth on failure and simpler routing.
• VL2 uses randomization to deal with volatility and churn in the data center traffic matrix. Its use of VLB spreads traffic across multiple paths irrespective of the destination.
• VL2 is based on existing routing and forwarding technologies (e.g., OSPF, ECMP etc.) to ensure that it can be implemented on existing hardware. Note that, OSPF is used to maintain switch-level topology, not to disseminate end host information.
• Names and locators are separated into application-specific addresses (AAs) and location-specific addresses (LAs). It should be obvious that AAs are fixed for entities, while LAs change with location changes.
• Finally, there is a two-tiered (one for performance and the other for consistency requirements) directory service that keeps track of the MAC, AA, and LA mappings for routing and forwarding purposes.

Discussion

The nicest part of this paper is the study (the first of its kind, as the authors put it) of data center traffic over a long period time. The observations are interesting, even though it might be possible that they are tightly coupled with the type of services running on that particular data center. Unlike the Internet, different data centers might have niche applications/services; it will be interesting to see if more such studies come out from different corporations.

Use of established protocols and ideas is another strong point of VL2. The authors synthesize the architecture using VLB, link state routing, ECMP, the concept of the separation of location and identity with Clos network topology (which apparently has been in use in telecom networks for a while and deals with the over-subscription problems). The evaluation section of the paper is also covers a lot of ground. There is one interesting thing regarding fairness though: the authors use Jain’s fairness index to show how evenly flows are split, and they rely on TCP for rate-limiting fair-sharing (max-min fairness supposedly). Can these two go together ? (not sure, can be further discussed)

There are lots of similarities between VL2 and PortLand along with some differences. But Amin Vahdat points out in his blog that the core difference is the philosophy of modifying the end hosts (in VL2) vs modifying the switches/routers (in PortLand); everything else stem from this. There is a nice article on network world that covers the bases of both the architectures. In addition to that, there is the difference between multi-rooted fat tree topology and Clos network topology; how different are they is not very clear though.