This tutorial will show you how to automate the creation of a simple Kafka cluster using Scalr Orchestration. The cluster will be fault tolerant, scalable, and highly available.
Apache Kafka is an open-source message broker project developed by the Apache Software Foundation written in Scala. The project aims to provide a unified, high-throughput, low-latency platform for handling real-time data feeds. More about Kafka.
Software used to build our cluster
We will be using a couple of different technologies to build our Kafka cluster:
ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services. All of these kinds of services are used in some form or another by distributed applications. We will be using Zookeeper to make sure that only one Kafka Leader is active at a time. More about Zookeeper.
Scalr features used in this tutorial
For this tutorial to work, it is assumed that you have:
- A Scalr Account
- An configured Environment
- An imported vanilla Ubuntu 14.04 Role, or registered Image into your environment (if you use Hosted Scalr, or have sync shared roles, you can skip this step).
- Step 1 - Prepare Scripts
- Step 2 - Create Kafka role
- Step 3 - Create and Launch Kafka Farm
- Step 4 - Scale the Cluster