Kafka platform on Confluent for Kubernetes

Teams needed real-time event streaming and a clean path from Kafka topics into the lakehouse. Self-hosted, secured, observable. Running in staging, with a production rollout next.

I built a streaming platform on Confluent for Kubernetes in KRaft mode, so no ZooKeeper. OAuth and RBAC run through an external identity provider. A schema registry sits behind TLS, custom Connect images carry an Iceberg sink and Debezium CDC, and connectors write topics into the Iceberg catalog. Ingress, DNS, metrics, and Grafana dashboards came with it.

A self-hosted streaming platform that feeds the lakehouse: Kafka topics land as Iceberg tables. Running in staging, with production next.

I built and operate it: the cluster, the auth, the schema registry, the custom Connect images, and the sinks into the catalog.

• Confluent for Kubernetes in KRaft mode, no ZooKeeper.
• OAuth and RBAC through an external identity provider; schema registry behind TLS.
• Custom Kafka Connect images carrying an Iceberg sink and Debezium CDC.
• Connectors write topics straight into the Iceberg catalog the lakehouse reads.
• Ingress, DNS, metrics, and Grafana dashboards alongside it.

• End-to-end authenticated streaming running in staging.
• A clean path from a Kafka topic to a queryable Iceberg table.
• Production rollout is the next milestone.