Mesos is an interesting project that cut across different use cases – big data analytics, web development, continuous integration and so on. I learned about Mesos from big data perspective, running Spark job on it. But in reality it has much broader utility acrosss the whole IT.
What is Mesos? Traditionally hardware resources, servers, and then later virtual machine are the unit of computing in datacenters. Dockers – the lightweight containers becoming another standard computing unit. Developers package application into docker containers and then deploy to virtual servers. Virtual servers divided into different clusters by functionality, for example, Hadoop cluster, Jenkins cluster, Web cluster, databases, application servers and so on and by purpose in development process – Dev, QA, Staging, Production and so on.
Cluster based deployment.
There are several problems with this approach:
- Resource utilization – let say you anticipated demand for your services doubled, ops team provisioned many more services, and now this servers sitting idle waiting to be utilized.
- Complexity in case of failure of nodes in individual cluster – let say one node in web farm fail, there is no way to utilize right away underutilized node fror jenkins cluster. Failures makes static partitioning more complex to manage.
- Developers has to know on which environment, on which servers their application is running. How many servers, their capacity and so on. This makes deployment and maintenance much costly. I remember having a spreadsheet with each server IP address, what it does, how to restart applications, how to ssh and so on.
- Virtual Machine licensing cost – those underutilized server cost money to run
In order to solve this problem, reserchers from UC Berkeley, created resource management system , called Mesos. Ben Hindman, one of them, said – “We wanted people to be able to program for the datacenter just like they program for their laptop.” Interestingly enough, hugely successful open source project Spark was created as a test for Mesos.
Mesos tread all the datacenter resources as one big, giant computer where different applications running.
Apache Mesos abstracts CPU, memory, storage, and other compute resources away from machines (physical or virtual), enabling fault-tolerant and elastic distributed systems to easily be built and run effectively. Instead of focusing on one application running on a specific server, Mesos’ resource isolation allows for multi-tenancy – the ability to run multiple applications on a single machine – leading to more efficient utilization of computing resources.
It also provides fault-tolerance and high-availability out of the box. Where you run different application on different clusters, you now run all of them on one cluster and mesos intelligently distribute resource among them. Instead of paging sys admin for failing node, mesos will distribute the load to another nodes. It has built-in support for isolating processes in containers using Linux control groups (cgroups) and Docker containers, allowing multiple applications to run alongside each other on a single machine. For example, application need additional capacity, it many utilize machine that running jenkins job at the same time, provided there are sufficient resources available. Deployment process become much simpler – instead of deploying to a particular cluster, application get deployed to Mesos and it will handle resources for you. Mesos also allow easily build distributed system on top of it. It could take only 300-400 Scala code to create fully distributed, fault-tolerant system. Mesos can scale to 10,000 nodes, battle proven in production at Twitter, have vibrant community, commercial support from Mesosphere, runs on AWS, GCE and on premises.
Here is a list of companies who is using Mesos in production:
Mesos architecture consist of several components: Masters, slaves, Frameworks and Zookeeper. Let’s review role of the each components:
Framework – Distributed applications that run on top of Mesos are called frameworks. Framework consist of two parts: scheduler – responsible for coordinating the execution and Executors – provides the ability to control the task execution. Executors can realize a task execution in many ways, like starting thread of execution.
Master – The Mesos master is primarily responsible for allocating resources to different frameworks and managing the task life cycle for them. Resources offer represents a unit of allocation in the Mesos world. It’s a vector of resource available on a node. For example, cpu and memory.
Slaves are responsible for executing tasks from frameworks using the resources they have.They also provide proper isolation while running multiple tasks.
Zookeper is a consensus service, responsible for electing a leader – master in the cluster.
This bring us to most important concepts in Mesos. They are:
- resource offers
- resource isolation
- two-tier scheduling
Resource offer. Each Mesos slave advertised it’s resources to the master. For example, A Mesos slave advertises to the leading Mesos master that it has 8 CPUs, 16 GB memory, and 64 GB disk available. The Mesos allocation module decides that the master should advertise the entire resource offer to Framework A’s scheduler. Framework A’s scheduler only accepts half of the resources offered to it, leaving 4 CPUs, 8 GB memory, and 32 GB disk available for other applications.
The Mesos allocation module decides that the master should advertise all of the remaining (unallocated) resources in the offer to Framework B’s scheduler. One of the examples of frameworks could be Apache Spark and Marathon.
Mesos Resource Offer
Resource Isolation. Second concept is resource isolation. How Mesos are able to run completly different software on one node? Mesos has built-in support for isolating processes in containers using Linux control groups (cgroups) and Docker containers, allowing multiple applications to run alongside each other on a single machine. Mesos also support custom containerizers as well.
Last concept is Resource allocation. The Mesos master gets information on the available resources from the Mesos slaves,and based on resource policies, the Mesos master offers these resources to different frameworks. Mesos is using Weighted Dominant Resource Fairness algorithm to decide which framework which resources get offered. For details, please check “Dominant Resource Fairness: Fair Allocation of Multiple Resource Types” Why it is weigthed? Because you can provide weight to influence allocation decision, for example: –weights prod:30, QA:20, dev:10. This way we allocated 30% of resources to prod and 20% to QA.
Mesos allow to run many applications without any changes. But one thing it’s running spark and some other Big Data tools that already distributed and aware of Mesos, and another is our web services. So the question is How do I run my microservices (SOA) on Mesos? And the answer is simple. You don’t have to change your application. All we need is to use meta-framework like Marathon. Marathon is a framework used for running long-running services on Mesos.
These services have high availability requirements, which means that Marathon should be able to start the service instance
on other machines in case of a failure and should be able to scale elastically. While Mesos provide out of the box isolation level with cgroup, I would suggest wrapping microservices in dockers container.
Mesos has several interesting applications, for example, Chronos and Aurora. Chronos is distributed, fault-tolerant scheduler.
Typical jobs include backups, Extract-Transform-Load (ETL) jobs, running other frameworks, and so on.
Chronos allows you to run shell scripts and also supports dependencies and retries. Aurora runs applications and services across a shared pool of machines, and is responsible for keeping them running, forever. When machines experience failure, Aurora intelligently reschedules those jobs onto healthy machines.
What type of application are good candidate for Mesos?
Stateful, or monolithic, applications that need to persist data to disk are not good candidates for running on Mesos
How to start. The easier way to start with Apache Mesos is to use Vagrant and Virtual Box. Vagrant ready images are available. One of them is playa-mesos. Playa Mesos helps you quickly create Apache Mesos test environments. This project relies onVirtualBox, Vagrant, and an Ubuntu box image which has Mesos and Marathon pre-installed.
Mesosphere – commercial company, provides a lot of good tutorial for Mesos.