Distributed in-memory datasets

AMPLab, "Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing," UCB/EECS-2011-82, 2011. [PDF]

Russell Power, Jinyang Li, "Piccolo: Building Fast, Distributed Programs with Partitioned Tables," OSDI, 2010. [PDF]

Summary

MapReduce and similar frameworks, while widely applicable, are limited to directed acyclic data flow models, do not expose global states, and generally slow due … Continue Reading ››

Cloud databases

MIT, "Relational Cloud: A Database-as-a-Service for the Cloud," CIDR, 2011. [PDF]

Divyakant Agrawal, Amr El Abbadi, Sudipto Das, Aaron J. Elmore, "Database Scalability, Elasticity, and Autonomy in the Cloud," DASFAA, 2011. [PDF]

Relational Cloud

The key idea of the Relational Cloud project is to define the concept of transactional Database-as-a-Service (DBaaS), identify the key challenges toward … Continue Reading ››

Declarative and finite state machine approaches to Cloud programming

Perter Alvaro, Tyson Condie, Neil Conway, Khaled Elmeleegy, Joseph M. Hellerstein, Russell Sears, "BOOM Analytics: Exploring Data-Centric, Declarative Programming for the Cloud," EuroSys, 2010. [PDF]

Joe Armstrong, "Erlang: A Survey of the Language and Its Industrial Applications," Ninth Exhibition and Symposium on Industrial Applications of Prolog, 1996. [PDF]

BOOM

BOOM or Berkeley Orders-Of-Magnitude adopts a … Continue Reading ››

Don’t Settle for Eventual: Scalable Causal Consistency for Wide-Area Storage with COPS

Wyatt Lloyd, Michael J. Freedman, Michael Kaminsky, and David G. Andersen, "Don’t Settle for Eventual: Scalable Causal Consistency for Wide-Area Storage with COPS," SOSP, 2011. [PDF]

Summary

This paper introduces a new consistency model, causal+, that extends the causal consistency model and lies between sequential and causal consistency models. The authors claim that causal+ is the … Continue Reading ››

PNUTS: Yahoo!’s Hosted Data Serving Platform

Yahoo! Research, "PNUTS: Yahoo!’s Hosted Data Serving Platform," PVLDB, 2008. [PDF]

Summary

PNUTS is a scalable, highly available, and geographically distributed (but low latency) data store used by most Yahoo! online properties. To achieve both availability and partition tolerance, it uses a novel notion of consistency called per-record timeline consistency; under this model, all replicas of … Continue Reading ››

Data-parallel pipelines using high-level languages

Microsoft, "DryadLINQ: A System for General-Purpose Distributed Data-Parallel Computing Using a High-Level Language," OSDI, 2008. [PDF]

Google, "FlumeJava: Easy, Efficient Data-Parallel Pipelines," PLDI, 2010. [LINK]

Background

Data-parallel computing systems expose high-level abstractions to the users to reason about distributed computations, while handling low-level tasks of scheduling and automated fault-tolerance without any user input. At … Continue Reading ››

Dremel: Interactive Analysis of Web-Scale Datasets

Google, "Dremel: Interactive Analysis of Web-Scale Datasets," VLDB, 2010. [PDF]

Summary

Dremel is Google's interactive  ad hoc query system for analysis of read-only nested data. Unlike MapReduce, Dremel is aimed toward data exploration, monitoring, and debugging, where near real-time performance is of utmost importance. To achieve scalability and performance, Dremel builds upon three key ideas:
  1. It … Continue Reading ››

Dynamo: Amazon’s Highly Available Key-value Store

Amazon, "Dynamo: Amazon's Highly Available Key-value Store," SOSP, 2007. [PDF]

Summary

Dynamo is a highly available (99.9th percentile) key-value storage mechanism that sacrifices traditional consistency models for eventual consistency to achieve availability. Dynamo works with a simple query model, where read/write (get() and put()) operations are performed on data items uniquely identified by their keys. … Continue Reading ››

Bigtable: A Distributed Storage System for Structured Data

Google, "Bigtable: A Distributed Storage System for Structured Data," OSDI, 2006. [PDF]

Summary

Bigtable is a large-scale (petabytes of data across thousands of machines) distributed storage system for managing structured data. It is built on top of several existing Google technology (e.g., GFS, Chubby, and Sawzal) and used by many of Google's online … Continue Reading ››

SCADS: Scale-Independent Storage for Social Computing Applications

Michael Armbrust, Armando Fox, David A. Patterson, Nick Lanham, Beth Trushkowsky, Jesse Trutna, Haruki Oh, "SCADS: Scale-Independent Storage for Social Computing Applications," CIDR, 2009. [PDF]

Summary

SCADS (Scalable Consistency Adjustable Data Storage) is a proposal for a collection of components leveraging database, control theory, and machine learning techniques to achieve data scale independence for rapidly … Continue Reading ››