Data Platform Engineer · SF Bay Area
Jordan Allen Lewis
Taking data platforms from proof of concept to production. Lately that's meant a federated lakehouse on Dremio, a Kafka streaming platform, and AI tooling, all on Kubernetes, backing a customer-facing product and about a dozen internal teams.
What I build, and why
I build and run data platforms: a federated lakehouse that queries 40 systems as if they were one, plus streaming, a governance catalog, an API layer, and AI tooling on top.
The goal is simple: let a business use all of its data to make decisions and power products, without first copying it into one warehouse or waiting on the data team to build a pipeline.
- 40
- Sources, one engine
- 30K+
- Datasets, federated
- 99.9%
- Platform uptime
Selected work
Things I've built
- 01
Platform
Production data lakehouse on Dremio + Kubernetes
A production-critical service at 99.9% uptime. It powers a customer-facing SaaS product, and a dozen internal teams, data scientists, and engineers build on it directly, self-serve.
DremioApache IcebergKubernetesHelmHDFSSnowflakeGrafana - 02
Streaming
Kafka platform on Confluent for Kubernetes
Self-hosted, secured real-time data streaming, running in staging. The streamed data lands straight in the lakehouse, ready to query.
Apache KafkaConfluentKRaftOAuthDebeziumApache Iceberg
Writing
Latest thinking
Databricks Summit: Three Things That Tested Our Roadmap
I run an Iceberg lakehouse on Dremio, a Kafka platform, and an AI data steward I built, and none of it is Databricks. I went to their summit to pressure-test our roadmap, and three companies on stage kept pointing at the same shape: the engine is the swappable part, the open layer underneath is the bet that lasts.
Read more
Get new posts by email
Occasional, only when I publish something. Data platforms, lakehouses, streaming, and where AI earns its keep in real pipelines.
Let's talk.
Happy to talk data platforms, lakehouses, or where AI actually earns its keep in infrastructure. LinkedIn is the fastest way to reach me.