Apache Kafka Series - Kafka Monitoring & Operations

Learn how AWS CloudFormation lets you declaratively define infrastructure to quickly set up a Kafka cluster. Templates automate the creation of security groups, EC2 instances, load balancers, and S3 resources.

Spin up a three-availability-zone kafka cluster with a zookeeper server and three kafka brokers, plus an administration and monitoring server via four cloud formation templates.

Deploy a kafka cluster by applying four cloudformation templates for zookeeper resources, zookeeper compute, kafka resources, and kafka compute, while configuring ssh keys and subnets.

Deploy a three-broker kafka cluster with cloudformation by uploading a template, configuring subnets and IPs, and launching the kafka resources stack to create a running cluster.

Explore our zookeeper cluster hands-on: SSH into zookeeper and kafka nodes, inspect systemd services and kafka binaries, view logs with journalctl, and review cloud formation templates that define cluster.

Perform a hands-on setup of a three-broker Kafka cluster, configure properties, connect to Zookeeper, create a dummy topic with replication factor three and twelve partitions, and run producer and consumer.

Stop all cluster machines to save overnight costs, then restart in steps: start ZooKeeper first, verify status, then start Kafka and verify with a console consumer on dummy_topic.

Learn to install and configure a visual Kafka administration suite by deploying Zoom Navigator, Kafka Manager, and Kafka Monitor on an Amazon Linux 2 instance using Docker and Docker Compose.

Learn to set up Zoo Navigator with Docker Compose, test zookeeper connectivity, and run as a systemd service for easy startup. Configure ports and access via the UI.

Set up the Kafka Manager UI with Docker Compose, configure zookeeper, and verify brokers and topics, then enable Kafka Manager as a systemd service.

Set up and run a Kafka monitor UI from source, configure zookeeper and bootstrap servers, deploy as a systemd service, and validate real-time producer-consumer metrics in the UI.

Learn how to monitor a Kafka cluster by setting up Prometheus and Gryphon to collect JMX metrics from Kafka and Zookeeper. Visualize end-to-end latency and metrics on the Dragana dashboard.

Set up the gem exporter agent on kafka brokers and install Prometheus on the administration machine, configure Prometheus to scrape Kafka metrics, and verify with curl.

Set up two additional Kafka brokers with Prometheus by installing the Java agent and updating Prometheus targets for brokers 2 and 3.

Learn to set up zookeeper with our Gen X exporter agent and Prometheus, configure a zookeeper yaml, enable extra_args, verify metrics via curl, and integrate zookeeper metrics into Prometheus.

Set up Grafana on the administration server using Prometheus as the data source, enable anonymous access, run Grafana as a systemd service, and verify port 3000 for Kafka monitoring.

Set up a Grafana Kafka dashboard by importing a prebuilt dashboard and configuring a Prometheus data source on localhost to monitor cpu, memory, and per-topic bytes in/out.

Monitor key kafka metrics: one active controller, zero under replicated partitions, and zero offline partitions; set alerts and reference kafka and confluent docs on yammer metrics and gen-x.

Learn to build per-broker kafka metrics dashboards in grafana by editing panel formulas, adjusting legends, and using prometheus data sources to visualize bytes in/out and broker counts.

Add and duplicate Grafana panels for Kafka metrics from Prometheus, including online partitions, unclean leader election rates, under replicated partitions, and offline partitions.

Experiment with killing a Kafka broker to observe Grafana dashboards update in real time, showing online and offline partitions, under replicated partitions, and broker recovery through monitoring metrics.

Apply pointers to make Zookeeper and Kafka production-ready through automation and secure administration tools. Plan dedicated monitoring hardware with Prometheus and Gryphon, and automation with systemd, Terraform, Puppet, or Kubernetes.

Learn automated Kafka broker rolling restarts with Kafka utils, install Jolokia on broker 1, and configure Wallachia and Troika agents to monitor metrics and ensure under replicated partitions equal zero.

Install Jolokia on broker two and broker three, configure java agents, restart Kafka, and verify with curl and jq; ensure admin can query all brokers and run with Prometheus exporter.

Install the Yelp Kafka utils on the admin server, generate and publish ssh keys to the brokers, and configure the Kafka yaml for a zookeeper-backed cluster's rolling restart.

Run the Kafka rolling restart script to safely restart brokers and verify stability. Configure cluster type and start/stop commands, monitor under replicated partitions and missing brokers during the process.

Update Kafka configurations by applying min.in.sync.replicas=2 across all brokers using a rolling restart, validating changes with journalctl and grep, one broker at a time.

Apply the second method to update Kafka configurations without restarting brokers using the kafka-configs command. See how Zookeeper-stored dynamic configs override server.properties with per-broker, cluster-wide, and default scopes.

Learn how to rebalance kafka partitions manually using plain json, zookeeper and cli tools. Then verify with kafka manager and perform preferred replica election.

Learn to reassign partitions in Kafka Manager, generate or manually assign partitions across brokers, run reassignments and preferred replica elections via a user interface for quick, reliable partition rebalancing.

Learn to rebalance a Kafka cluster using LinkedIn's Kafka tools, including generating and executing partition reassignment with the signer, balancing partitions across brokers, and handling leadership.

Demonstrate increasing a Kafka topic replication factor with plain commands and JSON, including topic creation with partitions, partition reassignment, and verification of data integrity.

Increase topic replication factors with LinkedIn's set replication factor tool, run a dry run with the signer, then execute to raise to three across topics including Kafka monitor.

Add a fourth Kafka broker via cloud formation, configure monitoring with Prometheus, verify registration in Kafka manager. Note that the new broker has no partitions yet.

Move partitions to broker four using Kafka reassignment commands, performing six batches of ten moves to evenly rebalance 60 partitions, then verify with a topic describe command and the UI.

Replace a Kafka broker while keeping the EBS volume, upgrading the instance type, and reattaching the volume to the new broker to maintain data and avoid disruption to the cluster.

Replace a broker after losing an EBS volume by stopping it, wiping data, restarting, and letting Kafka replication recover missing data with a replication factor of two or three.

Learn to safely remove a broker by moving all partitions off that broker, then terminating it, avoiding premature deletion and zookeeper changes.

Remove broker four by reassigning its 60 partitions to other brokers using the LinkedIn Kafka tools, then terminate the broker via CloudFormation and delete the EBS volume and IP resources.

Upgrade a Kafka cluster from 1.1 to 2.0 with a five-step rolling restart, updating inter-broker and log versions, binaries, and the protocol version, while noting up and down conversions.

Learn how kafka brokers use inter broker protocol versions for upgrades, including how zero copy and up and down conversions impact memory and client compatibility.

Execute a step-by-step upgrade from kafka 1.1 to 2.0, updating inter broker and log message format versions, then perform a rolling restart to apply changes with no downtime.

Upgrade kafka binaries from 1.1 to 2.0 by upgrading brokers one at a time, shutting down, updating, and restarting. Verify 2.0.0 with a version check and perform a rolling restart.

Apply step 3 by upgrading the inter broker protocol version to 2.0 across all Kafka brokers, restart them, and verify the change via logs and version checks.

Upgrade all kafka clients to a compatible version and update the log format to 2.0 on brokers after a rolling restart, following version-specific steps from zero 11 backward.

Upgrade the log message format version to 2.0 on all brokers, perform a rolling restart, and monitor with Kafka manager to ensure online partitions and stable CPU usage.

Learn how to clean up aws resources after this kafka monitoring course by stopping servers and deleting compute and zookeeper stacks to avoid ongoing charges.

Congrats on completing the course; implement and automate what you learned, practice topic and partition operations in a development cluster before production, and explore Prometheus and Gryphon on GitHub.

Celebrate completing this Apache Kafka series - Kafka monitoring & operations course by leaving a review with a star rating on Udemy to help future students.