Kafka within OpenShift

This is an outline of Apache Kafka as a system, and of running a cluster on OpenShift with the Strimzi operator. It is generic study notes. It is not a write-up of a cluster I operate, and it does not describe an employer's system.

The language notes next to this one are modern Java and Go or Rust. They are a separate thread.

Where a line says TODO, that part is still to be written from the docs, in my own words.

Why this note exists

I want to be able to explain the architecture, a consumer group, a connector, and the OpenShift custom resources without the docs open.

TODO: turn that into a short list of questions I can answer out loud when this note is finished.

Brokers, topics, and partitions

A Kafka cluster is a set of brokers. Producers append records to topics. A topic is split into partitions, and each partition is an ordered log. A record in a partition has an offset that increases along that partition.

Ordering is per partition. Records on different partitions have no order relative to each other. A key chooses the partition, so records with the same key stay in one partition and keep their order. A record with no key is spread across partitions.

TODO: a sketch of one topic with three partitions, and which keys land together.

TODO: write the ordering rule in one sentence I would actually say.

Replication

Each partition has a leader and zero or more followers. The in-sync replicas are the followers that have caught up with the leader. A record is available to consumers once it has reached the high watermark, the offset through which every in-sync replica has the record.

acks on the producer, min.insync.replicas, and whether an unclean leader election is allowed are the three settings that decide what happens when a broker is lost. They trade durability against availability.

TODO: write those three in my own words, each with the failure it accepts. Take the definitions from the Kafka documentation, and link the page.

KRaft

Kafka can store its own cluster metadata with KRaft, a Raft quorum of controllers, instead of an external ZooKeeper ensemble. Apache Kafka 4.0 is the first major release that only runs in KRaft mode. ZooKeeper mode was removed in that release. The 3.9 line is the last 3.x release, and the bridge used to migrate a ZooKeeper cluster before moving to 4.0.

TODO: from the upgrade notes, write what an operator still runs in KRaft mode, and the order of a migration off ZooKeeper. Link the page. Do not reconstruct the steps from this paragraph.

Producers

A producer batches records, chooses a partition (by key, or across partitions when there is no key), and waits according to acks before treating the send as done.

The idempotent producer and transactions are the two features I want a paragraph on. The idempotent producer avoids duplicates from retries. Transactions let a producer write to several partitions and commit them as one unit, at a cost in latency and complexity.

TODO: those two paragraphs, after reading the producer docs, plus a sentence on when I would leave transactions off.

Consumers and consumer groups

A consumer reads a partition from an offset. A consumer group divides the partitions of its subscribed topics across its members, so each partition is read by one member of the group at a time. When membership changes, the group rebalances and partitions move.

Offsets are how the group remembers its place. Committing an offset before the work is finished means a crash loses that work. Committing after the work means a crash can deliver the same records again. Consumers are expected to tolerate the duplicate.

TODO: enable.auto.commit versus committing after the work, in my own words, with the failure of committing too early.

TODO: one paragraph on cooperative and eager rebalancing, after the consumer docs. Name the protocol version I read about, including the next-generation rebalance protocol if that is the current one, and link it.

Kafka Connect and connectors

Kafka Connect is a separate runtime for moving data in and out of Kafka. A source connector writes into Kafka. A sink connector writes out of it. Connect workers can run as their own cluster, with a config topic, an offset topic, and a status topic.

A connector is the configured job. A task is a unit of that job, and a connector can have several tasks. A converter turns the record's payload into bytes or back, for example JSON or Avro. Those three are different things, and it is easy to talk about them as if they were one.

TODO: write connector, task, and converter in one sentence each, from the Connect docs.

TODO: name one source and one sink I want to be able to configure, after I have picked them. Leave the names out until then.

Kafka on OpenShift

Strimzi is a Kubernetes operator for Apache Kafka. On OpenShift the cluster is declared as custom resources, and the operator reconciles them into pods, services, and configuration. Red Hat's distribution of that operator was AMQ Streams, and has been renamed Streams for Apache Kafka. The rename is recorded in the Streams for Apache Kafka 2.9 release notes. Older pages still say AMQ Streams. The custom resources are the Strimzi ones either way.

The resources this note should be able to tell apart:

• Kafka is the cluster: brokers, and the controller quorum. KRaft or, on older versions, ZooKeeper is a setting of this resource.
• KafkaTopic is a topic, so topic configuration lives in the cluster's API rather than only in a kafka-topics command.
• KafkaUser is a user and its access control, when the user operator is enabled.
• KafkaConnect is a Connect cluster.
• KafkaConnector is a connector deployed onto that Connect cluster.

The topic operator and the user operator are switched on from the Kafka resource. Without them, KafkaTopic and KafkaUser are not reconciled.

A connector can also be created by posting its configuration to the Connect REST API. KafkaConnector is the operator's way of holding that configuration as a resource. They end up as the same connector inside Connect. The difference is what owns the configuration.

TODO: from the current Strimzi docs, summarise a minimal Kafka resource and the listener types I would actually choose on OpenShift. Link the doc. Keep the summary short enough that I can redraw it.

TODO: write how the topic operator and the user operator are enabled, from the docs, in a few lines.

TODO: say which I would use, KafkaConnector or the REST API, and why, after reading both.

TODO: storage, metrics, and what a rolling restart of brokers does to clients. Three short paragraphs, each from a doc page I link.

Open questions

TODO: questions go here as they come up. None yet beyond the placeholders above.