This tutorial will show you how to automate the creation of a fully fledged Mesos cluster using Scalr Orchestration. The cluster will be fault tolerant, scalable and highly available. It will include Mesos DNS to help you lookup where your Mesos processes are running, as well as Chronos for running scheduled tasks across your cluster. Our cluster will be setup to run Docker containers which can contain anything we want, from web applications to databases.
Mesos is a cluster framework that distributes processes across your server nodes. You can think of Mesos as one unified Linux Kernel across several servers. More about Mesos.
If you are not familiar with Docker, it's basically a light-weight "virtual server" that runs as a single process. These processes are called "containers" and can contain anything from a single binary to a fully fledged linux system. This will let you run the different components of your software stack in different containers, completely separated from each other. You can have one container for your app, one container for you memcache, one container for your database and so on. We will use Mesos to be able to distribute these containers across our cluster, as well as easily scaling them both vertically (giving more CPU/RAM to a single container) and horizontally (spreading clones of the container across different nodes). More about Docker.
Software used to build our cluster
We will be using a couple of different technologies to build our Mesos cluster:
Mesos is built using the same principles as the Linux kernel, only at a different level of abstraction. The Mesos kernel runs on every machine and provides applications (e.g., Hadoop, Spark, Kafka, Elastic Search) with API’s for resource management and scheduling across entire datacenter and cloud environments. Mesos provides a nice web UI where you can see your containers running.
ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services. All of these kinds of services are used in some form or another by distributed applications. We will be using Zookeeper to make sure that only one Mesos Master is active at a time. For fault tolerance, we will be running 3 Mesos Masters. One of them will be the active master and the other ones will be stand-by masters. If the active master goes down, Zookeeper makes sure that another node is elected as the new active master. More about Zookeeper.
Marathon is run on top of Mesos to provide Docker support and fault tolerance. If a Docker process dies (because of a software or hardware fault), Mesos will notice this and reschedule containers on healthy nodes. It comes with a nice web UI and REST API. More about Marathon.
Chronos is a fault tolerant job scheduler for Mesos, and replacement for Cron. It comes with a nice web UI and REST API. More about Chronos.
Mesos DNS is a simple DNS server that makes it easy to keep track of what nodes your Docker containers run on. If you deploy 10 instances of a Docker container running Apache, the DNS server allows you to lookup the IPs of those 10 instances. Mesos DNS keeps track of Docker containers as they move around your cluster and updates records accordingly. More about Mesos DNS.
Scalr features used in this tutorial
For this tutorial to work, it is assumed that you have:
- A Scalr Account
- An configured Environment
- An imported vanilla CentOS 7, RedHat 7 or Ubuntu 14.04 Role, or registered Image into your environment (if you use Hosted Scalr, or have sync shared roles, you can skip this step).
- Step 1 - Prepare Mesos Scripts
- Step 2 - Create Mesos Role
- Step 3 - Create Mesos Farm
- Step 4 - Scale up your cluster
- Step 5 - Deploy and scale Docker Containers
- Step 6 - Deploy containers with contraints