Since the inception of coflow in 2012, the abstraction and works surrounding it are growing at a fast pace. In addition to systems building, we have seen a rise of theoretical analyses of the coflow scheduling problem. One of the most recent ones to this end has even received a Best Student Paper … Continue Reading ››
Tag Archives: Coflow
Carbyne Accepted to Appear at OSDI’2016
Update: Camera-ready version is available here now!
With the wide adoption of distributed data-parallel applications, large-scale resource scheduling has become a constant source of innovation in recent years. There are tens of scheduling solutions that try to optimize for objectives such as user-level fairness, application-level performance, and cluster-level efficiency. However, given the well-known tradeoffs between fairness, performance, and efficiency, these solutions have traditionally focused … Continue Reading ››
CODA Accepted to Appear at SIGCOMM’2016
Update: Camera-ready version is available here now!
Since introducing the coflow abstraction in 2012, we've been working hard to make it practical one step at a time. Over the years, we've worked on efficient coflow scheduling, removed clairvoyance requirements in coflow scheduling, and performed fair sharing among coexisting coflows. Throughout all these efforts, one requirement remained constant: all … Continue Reading ››
HUG Accepted to Appear at NSDI’2016
Update: Camera-ready version is available here now!
With the advent of cloud computing and datacenter-scale applications, simultaneously dealing with multiple resources is the new norm. When multiple parties have multi-resource demands, fairly dividing these resources (for some notion of fairness) is a core challenge in the resource allocation literature. Dominant Resource Fairness (DRF) in NSDI'2011 was the first work to … Continue Reading ››
Aalo Accepted to Appear at SIGCOMM’2015
Update: Camera-ready version is available here now!
Last SIGCOMM we introduced the coflow scheduling problem and presented Varys that addressed its clairvoyant variation, i.e., when all the information of individual coflows are known a priori, and there is no cluster and task scheduling dynamics. In many cases, these assumptions do not hold very well and left us with two primary … Continue Reading ››
Orchestra is the Default Broadcast Mechanism in Apache Spark
With its recent release, Apache Spark has promoted Cornet—the BitTorrent-like broadcast mechanism proposed in Orchestra (SIGCOMM'11)—to become its default broadcast mechanism. It's great to see our research see the light of the real-world! Many thanks to Reynold and others for making it happen.
MLlib, the machine learning library of Spark, will enjoy the biggest boost from this change because of the broadcast-heavy nature of … Continue Reading ››
Varys Developer Alpha Released!
We are glad to announce the first open-source release of Varys, an application-aware network scheduler for data-parallel clusters using the coflow abstraction. It's a stripped-down dev-alpha release for the experts, so please be patient with it!
A quick overview of the system can be found at varys.net. Here is a 30-second summary:
Varys is an open … Continue Reading ››
Varys accepted at SIGCOMM’2014: Coflow is coming!
Update 2: Varys Developer Alpha released!
Update 1: Latest version will be is available here soon now!
We introduced the coflow abstraction back in 2012 at HotNets-XI, and now we have something solid to show what coflows are good for. Our paper on efficient and deadline-sensitive coflow scheduling has been accepted to appear at this year's … Continue Reading ››
Sinbad accepted to appear at SIGCOMM’2013
Update: Latest version is available here!
Our paper on leveraging flexibility in endpoint placement, aka Sinbad/Usher/Orchestrated File System (OFS), has been accepted for publication at this year's SIGCOMM. We introduce constrained anycast in data-intensive clusters in the form of network-aware replica placement for cluster/distributed file systems. This is the second piece of my dissertation research, … Continue Reading ››
I’m a PhD candidate now!
I've finally taken the qualifying exam this morning and presented my proposal in front of professors Sylvia Ratnasamy, Ion Stoica, Scott Shenker, and Marti Hearst. It was good to hear them say I passed :)
My proposal was on using coflows to better manage the network for data-intensive cluster applications: how to manage each one individually, how … Continue Reading ››