How you can Hyperscale your Applications Using Mesos & Marathon

In a previous blog post we have seen what Apache Mesos is and how it helps to create dynamic partitioning of our available resources which results in increased utilization, efficiency, reduced latency, and better ROI. We also discussed how to install, configure and run Mesos and sample frameworks. There is much more to Mesos than above.
In this post we will explore and experiment with a close to real-life Mesos cluster running multiple master-slave configurations along with Marathon, a meta-framework that acts as cluster-wide init and control system for long running services. We will set up a 3 Mesos master(s) and 3 Mesos slaves(s), cluster them along with Zookeeper and Marathon, and finally run a Ruby on Rails application on this Mesos cluster. The post will demo scaling up and down of the Rails application with the help of Marathon. We will use Vagrant to set up our nodes inside VirtualBox and will link the relevant Vagrantfile later in this post.
To follow this guide you will need to obtain the binaries for

• • Ubuntu 14.04 (64 bit arch) (Trusty)
  • Apache Mesos
  • Marathon
  • Apache Zookeeper
  • Ruby / Rails
  • VirtualBox
  • Vagrant
  • Vagrant Plugins
    • vagrant-hosts
    • vagrant-cachier

Let me briefly explain what Marathon and Zookeeper are.
Marathon is a meta-framework you can use to start other Mesos frameworks or applications (anything that you could launch from your standard shell). So if Mesos is your data center kernel, Marathon is your “init” or “upstart”. Marathon provides an excellent REST API to start, stop and scale your application.
Apache Zookeeper is coordination server for distributed systems to maintain configuration information, naming, provide distributed synchronization, and group services. We will use Zookeeper to coordinate between the masters themselves and slaves.
For Apache Mesos, Marathon and Zookeeper we will use the excellent packages from Mesosphere, the company behind Marathon. This will save us a lot of time from building the binaries ourselves. Also, we get to leverage bunch of helpers that these packages provide, such as creating required directories, configuration files and templates, startup/shutdown scripts, etc. Our cluster will look like this:

Installation

$ sudo apt-key adv --keyserver keyserver.ubuntu.com --recv E56151BF
$ echo "deb http://repos.mesosphere.io/ubuntu trusty main" |  sudo tee /etc/apt/sources.list.d/mesosphere.list
$ sudo apt-get -y update

On master nodes:

$ sudo apt-get install mesosphere

On slave nodes:

$ sudo apt-get install mesos

$ vagrant up

Configuration

Zookeeper

$ sudo service zookeeper stop

$ echo 10 | sudo tee /etc/zookeeper/conf/myid

# /etc/zookeeper/conf/zoo.cfg
server.10=10.10.20.11:2888:3888
server.30=10.10.20.12:2888:3888
server.50=10.10.20.13:2888:3888

All Nodes

Zookeeper Connection Details

#/etc/mesos/zk
zk://10.10.20.11:2181,10.10.20.12:2181,10.10.20.13:2181/mesos

IP Addresses

Masters

$  echo  | sudo tee /etc/mesos-master/ip
$ sudo cp /etc/mesos-master/ip /etc/mesos-master/hostname

Slaves

$ echo  | sudo tee /etc/mesos-slave/ip
$ sudo cp /etc/mesos-slave/ip /etc/mesos-slave/hostname

If you are using the Mesosphere packages, then you get a bunch of intelligent defaults. One of the
most important things you get is a convenient way to pass CLI options to Mesos. All you need to do is
create a file with same name as that of the CLI option and put the correct value that you want to
pass to Mesos (master or slave). The file needs to be copied to a correct directory. In case of Mesos
masters, you need to copy the correct file to /etc/mesos-master and for slaves you should copy the
file to /etc/mesos-slave For example: echo 5050 > sudo tee /etc/mesos-slave/port

Mesos Servers

$ sudo service mesos-slave stop

$ echo manual | sudo tee /etc/init/mesos-slave.override

Mesos Slaves

$ sudo service mesos-master stop
$ echo manual | sudo tee /etc/init/mesos-master.override
$ echo manual | sudo tee /etc/init/zookeeper.override

Marathon

$ sudo mkdir -p /etc/marathon/conf

$ sudo cp /etc/mesos-master/ip /etc/marathon/conf/hostname
$ sudo cp /etc/mesos/zk /etc/marathon/conf/master

$ echo zk://10.10.20.11:2181,10.10.20.12:2181,10.10.20.13:2181/marathon \
| sudo tee /etc/marathon/conf/zk

Starting Services

Master

$ sudo service zookeeper start
$ sudo service mesos-master start
$ sudo service marathon start

 

Slave

$ sudo service mesos-slave start

 

Running Your Application

Marathon provides nice Web UI to set up your application. It also provides an excellent REST API to create, launch, scale applications, check health status and more.
Go to your Marathon Web UI, if you followed the above instructions then the URL should be one of the Mesos masters on port 8080 ( i.e. http://10.10.20.11:8080 ). Click on “New App” button to deploy a new application. Fill in the details. Application ID is mandatory. Select relevant values for CPU, Memory, Disk Space for your application. For now let number of instances be 1. We will increase them later when we scale up the application in our shiny new cluster.
There are a few optional settings that you might have to take care depending on our your slaves are provisioned and configured. For this post, I made sure each slave had Ruby, Ruby related dependencies and Bundler gem were installed. I took care of this when I launched and provisioned the slaves nodes.
One of the important optional settings is “Command” that Marathon can execute. Marathon monitors this command and reruns it if it stops for some reason. Thus Marathon claims to fame as “init” and runs long running applications. For this post, I have used the following command (without the quotes).

“cd hello && bundle install && RAILS_ENV=production bundle exec unicorn -p 9999”

This command reads the Gemfile in the Rails application, installs all the necessary gems required for the application, and then runs the application on port 9999.
I am using a sample Ruby On Rails application. I have put the url of the tarred application in the URI field. Marathon understands a few archive/package formats and takes care of unpacking them so we needn’t worry about them. Applications need resources to run properly, URIs can be used for this purpose. Read more about applications and resourceshere.
Once you click “Create”, you will see that Marathon starts deploying the Rails application. A slave is selected by Mesos, the application tarball is downloaded, untarred, requirements are installed and the application is run. You can monitor all the above steps by watching the “Sandbox” logs that you should find on Mesos main web UI page. When the state of task will change from “Staging” to “Running” we have a Rails application run via Marathon on a Mesos slave node. Hurrah!
If you followed the steps from above, and you read the “Sandbox” logs you know the IP of the node where the application was deployed. Navigate to the SLAVE_NODE_IP:9999 to see your rails application running.

Scaling Your Application

All good but how do we scale? After all, the idea is for our application to reach web scale and become the next Twitter, and this post is all about scaling application with Mesos and Marathon. So this is going to be difficult! Scaling up/down is difficult but not when you have Mesos and Marathon for company. Navigate to the application page on Marathon UI. You should see a button that says “Scale”. Click on it and increase the number to 2 or 3 or whatever you prefer (assuming that you have that many slave nodes). In this post we have 3 slave nodes, so I can choose 2 or 3. I chose 3. And voila! The application is deployed seamlessly to the other two nodes just like it was deployed to the first node. You can see for yourself by navigating to SLAVE_NODE_IP:9999 where SLAVE_NODE_IP will be the IP of the slave where the application was deployed. And there you go, you have your application running on multiple nodes.
It would be trivial to put these IPs behind a load-balancer and a reverse proxy so that access to your application is as simple as possible.

Graceful Degradation (and vice versa)

Sometimes nodes in your clusters go down for one reason or other. Very often we get an email from your IaaS provider that your node will be retired in few days time and at other times a node dies before you could figure out what happened. When such inevitable things happen and the node in question is part of the cluster running the application, the dynamic duo of Mesos and Marathon have your back. The system will detect the failure, will de-register the slave and deploy the application to a different slave available in the cluster. You could tie it up with your IaaS-provided scaling option and spawn required number of new slave nodes as a part of your cluster which once registered with the Mesos cluster can run your application.

Marathon REST API

Although we have used the Web UI to add a new application and scale it. We could have done the same (and much more) using REST API and thus do Marathon operations via some program or scripts. Here’s an simple example that will scale the application to 2 instances. Use any REST client or just curl to make a PUT request to the application ID, in our case http://10.10.20.11:8080/v2/apps/rails-app-on-mesos-marathon with the following JSON data as payload. You will notice that Marathon deploys the application to another instance if there was only 1 instance before.

{
    "instances" : 2
}

You can do much more than above, do health checks, add/suspend/kill/scale applications etc. This can become a complete blog post in itself and will be dealt at a later time.

Conclusion

Scaling your application becomes as easy as pressing buttons with a combination of Mesos and Marathon. Setting up a cluster can become almost trivial once you get your requirements in place and ideally automate the configuration and provisioning of your nodes. For this post, I relied on simple Vagrantfile and a shell script that provision the system. Later I configured the system by hand as per above steps. Using Chef or alike would make the configuration step a single command work. In fact there are a few open-source projects that are already very successful and do just that. I have played witheverpeace/vagrant-mesos and it is an excellent starting point. Reading the code from these projects will help you understand a lot about building and configuring clusters with Mesos.
There are other projects that do similar things like Marathon and sometimes more. I definitely would like to mention Apache Aurora and HubSpot’s Singularity.