Update: Camera-ready version is available here now!
Last SIGCOMM we introduced the coflow scheduling problem and presented Varys that addressed its clairvoyant variation, i.e., when all the information of individual coflows are known a priori, and there is no cluster and task scheduling dynamics. In many cases, these assumptions do not hold very well and left us with two primary … Continue Reading ››Category Archives: Recent News
Orchestra is the Default Broadcast Mechanism in Apache Spark
With its recent release, Apache Spark has promoted Cornet—the BitTorrent-like broadcast mechanism proposed in Orchestra (SIGCOMM'11)—to become its default broadcast mechanism. It's great to see our research see the light of the real-world! Many thanks to Reynold and others for making it happen.
MLlib, the machine learning library of Spark, will enjoy the biggest boost from this change because of the broadcast-heavy nature of … Continue Reading ››
Presented Varys at SIGCOMM’2014
Update: Slides from my talk are online now!
Just got back from a vibrant SIGCOMM! I presented our recent work on application-aware network scheduling using the coflow abstraction (Varys). This is my fifth time attending and third time giving a talk. Great pleasure as always in meeting old friends and making new ones! SIGCOMM was held at Chicago this … Continue Reading ››Varys Developer Alpha Released!
We are glad to announce the first open-source release of Varys, an application-aware network scheduler for data-parallel clusters using the coflow abstraction. It's a stripped-down dev-alpha release for the experts, so please be patient with it!
A quick overview of the system can be found at varys.net. Here is a 30-second summary:
Varys is an open … Continue Reading ››
Sinbad Merged with HDFS Trunk at Facebook!
Today marks the end of my stay at Facebook. Over the last eight months, on and off, I have re-implemented, tested, evaluated, reproduced the results, and merged Sinbad (SIGCOMM'13) with Facebook's internal HDFS trunk. Several of the future work promised in the paper have been executed in the process, with varying results. It has been quite a … Continue Reading ››
Varys accepted at SIGCOMM’2014: Coflow is coming!
Update 2: Varys Developer Alpha released!
Update 1: Latest version will be is available here soon now!
Presented Sinbad at SIGCOMM’2013
Few days ago, I presented our work on network-aware replica placement (Sinbad) at this year's SIGCOMM. This is my fourth time attending and second time giving a talk. Great pleasure as always!
SIGCOMM was held at Hong Kong this year; this was my first trip to that part of the world to. I also visited Beijing … Continue Reading ››
Sinbad accepted to appear at SIGCOMM’2013
Update: Latest version is available here!
Our paper on leveraging flexibility in endpoint placement, aka Sinbad/Usher/Orchestrated File System (OFS), has been accepted for publication at this year's SIGCOMM. We introduce constrained anycast in data-intensive clusters in the form of network-aware replica placement for cluster/distributed file systems. This is the second piece of my dissertation research, … Continue Reading ››I’m a PhD candidate now!
I've finally taken the qualifying exam this morning and presented my proposal in front of professors Sylvia Ratnasamy, Ion Stoica, Scott Shenker, and Marti Hearst. It was good to hear them say I passed :)
My proposal was on using coflows to better manage the network for data-intensive cluster applications: how to manage each one individually, how … Continue Reading ››
Coflow accepted at HotNets’2012
Update: Coflow camera-ready is available online! Tell us what you think!
Our position paper to address the lack of a networking abstraction for cluster applications, "Coflow: A Networking Abstraction for Cluster Applications," has been accepted at the latest workshop on hot topics in networking. We make the observation that thinking in terms of flows is … Continue Reading ››