01
Flagship platform
Production data lakehouse on Dremio + Kubernetes
A production-critical service at 99.9%+ uptime that a dozen teams across the company query directly.
DremioApache IcebergKubernetesHelmHDFSSnowflakeGrafana
Read the write-upData Platform Engineer · SF Bay Area
I build the infrastructure
behind analytics.
I'm Jordan, a data platform engineer. I run a data lakehouse and a Kafka platform on Kubernetes, and build the tooling that keeps the data on them reliable and easy to find.
99.9%+
Production lakehouse uptime
12+
Teams using the platform
30+
Data sources in one SQL engine
10+
Services I run on Kubernetes
What I do
Platforms, end to end
I work across the whole stack that makes analytics possible: storage, query engines, streaming, identity, and governance.
Data lakehouse platforms
I run Dremio and an Apache Iceberg catalog on Kubernetes. One SQL engine queries HDFS, Hive, object storage, Snowflake, Postgres, and Mongo together, with a semantic layer, SSO, and query acceleration on top.
Reliability & operations
I own the platform end to end. Multi-environment CI/CD, on-call, incident response, HA/DR planning, and the dashboards that keep a production service above three nines.
Streaming & AI tooling
Kafka on Confluent for Kubernetes feeds the lakehouse. On top of it I build LLM and MCP tooling that documents the catalog and answers questions about it from your editor.
Selected work
Things I've built
- 02
Streaming
Kafka platform on Confluent for Kubernetes
Self-hosted streaming with end-to-end auth. Topics land in the lakehouse as Iceberg tables.
Apache KafkaConfluentKRaftOAuthDebeziumApache IcebergRead the write-up - 03
AI / Platform
AI data steward + MCP server
Documents hundreds of datasets on its own and answers catalog questions from an engineer's editor.
PythonLLMMCPApache IcebergSupersetPrometheusRead the write-up
Writing
Recent posts
- · 3 min read
I Built an AI Data Steward. The Hard Part Wasn't the AI.
A pipeline that documents a data catalog with an LLM sounds like a prompt-engineering problem. Almost none of my time went to prompts. It went to making the boring parts trustworthy.
- · 3 min read
When Your Identity Provider Lies About Group Claims
We wired group-based admin access through OIDC, granted the right group, and nobody got in. The token was the problem, and no amount of server config could fix it.
- · 3 min read
Credential Vending Only Works If Your Storage Speaks STS
An Iceberg REST catalog can hand short-lived storage credentials to query engines. That feature quietly assumes your object store implements one specific AWS STS call. Ours didn't.
Let's talk.
Happy to talk data platforms, lakehouses, or where AI actually earns its keep in infrastructure. LinkedIn is the fastest way to reach me.