Uber open-sources Peloton, a unified Resource Scheduler

2 min read

Earlier this month, Uber open-sourced Pelton, a unified resource scheduler that manages resources across distinct workloads. Pelton, first introduced in November last year, is built on top of Mesos.

“By allowing others in the cluster management community to leverage unified schedulers and workload co-location, Peloton will open the door for more efficient resource utilization and management across the community”, states the Uber team.

Peloton is designed for web-scale companies such as Uber that consist of millions of containers and tens of thousands of nodes. Peloton comes with advanced resource management capabilities such as elastic resource sharing, hierarchical max-min fairness, resource overcommits, and workload preemption.

Peloton uses Mesos to aggregate resources from different hosts and then further launch tasks as Docker containers. Peloton also makes use of hierarchical resource pools to manage elastic and cluster-wide resources more efficiently.

Before Peloton was released, each workload at Uber comprised its own cluster which resulted in various inefficiencies. However, with Peloton, mixed workloads can be colocated in shared clusters for better resource utilization.

Peloton feature highlights

Elastic Resource Sharing: Peloton supports hierarchical resource pools that help elastically share resources among different teams.
Resource Overcommit and Task Preemption: Peloton helps with improving cluster utilization by scheduling workloads that use slack resources.
Optimized for Big Data Workloads: Support has been provided for advanced Apache Spark features such as dynamic resource allocation.
Optimized for Machine Learning: There is support provided for GPU and Gang scheduling for TensorFlow and Horovod.
High Scalability: Users can scale to millions of containers and tens of thousands of nodes.

“Open sourcing Peloton will enable greater industry collaboration and open up the software to feedback and contributions from industry engineers, independent developers, and academics across the world”, states the Uber team.