Matei Zaharia, Benjamin Hindman, Andy Konwinski, Ali Ghodsi, Anthony D. Joseph, Randy Katz, Scott Shenker, Ion Stoica, “The Datacenter Needs an Operating System,” USENIX HotCloud, (June, 2011). [PDF]

Summary

In recent years, many have pondered whether the datacenter is the new computer. This paper answers that question somewhat positively and goes one step forward by proposing operating system (OS) primitives for this new computer. Essentially, the authors want to have a common software layer to manage resources and to provide shared services for the datacenter as a whole. They expect four key primitives from this OS-like abstraction:

• Resource sharing across diverse applications (e.g., MapReduce, Dryad, GFS, Hadoop) and users.
• Data sharing between programs written by these users.
• Programming abstractions to address the datacenter as a whole.
• Debugging facilities throughout the software stack across the datacenter.

To this end, the authors survey a collection of existing work (many from the AMPLab) that partly address problems across all four dimensions. They put forth a vision for an OS that will allow users to build Hadoop-like systems in a week, share data and the datacenter resourcers across different programming models and users, and debug such new systems as a whole instead of logging into individual machines.

Critique

This paper makes a valid point that there is a need for ‘some’ software layer to manage and schedule resources in a datacenter. It is, however, not clear whether that software layer is an OS in the traditional sense. Specially, it is hard to tell whether such an OS should include programming abstractions and debuggers, or they should come as bundled software with it.

In general, this is a good paper to generate discussions in the community and to mine citations, which it will, but I think there is a need for a lot more back-and-forth discussions to flesh out what is expected of such an OS, what are the bare minimums to be expected to call it an OS, what will the hardware/software layers look like, and how they will interact.