|This tutorial is a work in progress. All content is intended for proof of concept purposes only and is not suitable for production use.|
This tutorial will show you how to automate the creation of a simple Kafka cluster using Scalr Orchestration. The cluster will be fault tolerant, scalable, and highly available.
Apache Kafka is an open-source message broker project developed by the Apache Software Foundation written in Scala. The project aims to provide a unified, high-throughput, low-latency platform for handling real-time data feeds. More about Kafka.
We will be using a couple of different technologies to build our Kafka cluster:
ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services. All of these kinds of services are used in some form or another by distributed applications. We will be using Zookeeper to make sure that only one Kafka Leader is active at a time. More about Zookeeper.
For this tutorial to work, it is assumed that you have: