Data Platform Engineer · SF Bay Area
Jordan Allen Lewis
Taking data platforms from proof of concept to production. Recently that's been a Dremio platform on Kubernetes: one query engine federating 40 data sources in place, backing a customer-facing product and about a dozen internal teams.
The platform I work on, by the numbers
Selected work
Things I've built
- 01
Platform
Production data lakehouse on Dremio + Kubernetes
A production-critical service at 99.9% uptime. It powers a customer-facing SaaS product, and a dozen internal teams, data scientists, and engineers build on it directly, self-serve.
DremioApache IcebergKubernetesHelmHDFSSnowflakeGrafana - 02
Streaming
Kafka platform on Confluent for Kubernetes
Self-hosted, secured real-time data streaming, running in staging. The streamed data lands straight in the lakehouse, ready to query.
Apache KafkaConfluentKRaftOAuthDebeziumApache Iceberg
Writing
Latest thinking
Databricks Summit: Three Things That Tested Our Roadmap
I run an Iceberg lakehouse on Dremio, a Kafka platform, and an AI data steward I built, and none of it is Databricks. I went to their summit to pressure-test our roadmap, and three companies on stage kept pointing at the same shape: the engine is the swappable part, the open layer underneath is the bet that lasts.
Read more
Get new posts by email
Occasional, only when I publish something. Data platforms, lakehouses, streaming, and where AI earns its keep in real pipelines.
Let's talk.
Happy to talk data platforms, lakehouses, or where AI actually earns its keep in infrastructure. LinkedIn is the fastest way to reach me.