Introducing Kafka-Kit: Tools for scaling Kafka | Datadog (opens in new tab)

Datadog’s “Kafka Kit” is a collection of operational tools designed to make Apache Kafka easier to scale and manage. The post argues that Kafka’s built-in administrative mechanisms become difficult to use safely as clusters grow, particularly when rebalancing partitions or adding and removing brokers. Kafka Kit automates these workflows while emphasizing balanced assignments, controlled changes, and operational visibility.

Why Kafka Scaling Becomes Difficult

• Growing Kafka clusters require frequent partition movement and broker rebalancing.
• Native Kafka reassignment workflows can involve large, complex JSON configurations.
• Poorly planned changes can create:
  • Uneven storage and traffic distribution
  • Excessive network and disk I/O
  • Overloaded brokers
  • Extended recovery times
• Operational changes must account for replication, leadership, broker capacity, and rack or availability-zone placement.

Kafka Kit’s Approach

• Kafka Kit provides reusable tooling for common Kafka administration tasks.
• The tools generate and apply partition assignments instead of requiring operators to construct them manually.
• Assignments can be optimized for more even distribution of:
  • Partitions
  • Replicas
  • Leaders
  • Storage and traffic
• The tooling is intended to support both routine balancing and larger cluster changes, such as adding or decommissioning brokers.

Safer Partition Reassignment

• Reassignments can be performed incrementally rather than moving all partitions at once.
• Changes can be throttled to limit their effect on production workloads.
• Operators can inspect proposed assignments before applying them.
• Controlled movement reduces the risk of saturating Kafka brokers, disks, or network links.
• The approach makes long-running migrations easier to monitor and interrupt if necessary.

Operating Kafka at Scale

• Datadog built the tools from its experience running Kafka as a critical part of its data infrastructure.
• At large scale, Kafka administration needs to be repeatable and automatable rather than dependent on manual intervention.
• Separating planning from execution allows teams to validate capacity and placement before changing the cluster.
• Standardized tooling also helps reduce the chance of configuration errors during high-risk maintenance operations.

Kafka Kit is most useful for teams operating Kafka clusters large enough that manual partition management is unreliable or disruptive. Automating assignment generation, throttling, validation, and broker lifecycle changes can make scaling more predictable and safer.