Data engineering and operations
The pipeline layer decides everything above it. If ingestion is flaky, the dashboards lie. If there are no lower environments, every change is a production change. I build data platforms with software engineering discipline: version control, CI/CD, tests, observability, and an on-call culture that treats data incidents like the outages they are.
What’s usually broken
- Dev is prod. No lower environments, so every experiment runs against the data the business depends on.
- The pipeline is a monolith. One end-to-end job that takes hours, fails as a unit, and can only be debugged by the person who wrote it.
- Nothing is versioned. SQL lives in dashboards, transformations live in heads, and rollback means remembering.
- Silence until the CFO notices. No observability, no data quality gates, no alerting. Problems surface downstream, in meetings.
- The migration never ends. A warehouse move that started 2 years ago and still runs both systems, paying for each.
What I do
- Platform foundations: lifecycle environments, CI/CD, infrastructure as code, RBAC. The AXS Phase 1 build, generalized.
- Pipeline engineering: decomposing monoliths into composable incremental pipelines on dbt and Airflow, with contracts and quality validation at the seams.
- Migrations: BigQuery migrations and warehouse moves planned around workload evidence, executed with reconciliation so the numbers match before anything gets turned off.
- DataOps: git-based workflow for data code, CI checks for quality and governance, progressive rollout, postmortems.
- Observability and data quality infrastructure: automated quality checks, freshness and volume monitoring, catalog and lineage so an incident’s blast radius is knowable.
- Resilience: disaster recovery and business continuity planning for the data estate, tested rather than assumed.
How it works
Foundation work runs as a program with your engineers, because the point is a capability your team operates, and my job includes making sure they can. Standalone projects (a migration, an observability rollout) run fixed-scope with dates you can hold me to.
Proof
At AXS I took an estate with no CI/CD, no lower environments, and blended dev and prod data to 4 lifecycle environments, Terraform-managed infrastructure, RBAC, and a working Bronze ingestion path, in under a year, while hiring the org that runs it. At TelevisaUnivision I modernized monolithic ETL onto dbt and Airflow as composable incremental pipelines. At Goldman I stood up the Kafka-based pipelines behind Apple Card’s transaction stream, with daily reconciliation to a regulated bank’s standard.
Start a conversation
If your team is afraid to deploy on Fridays, start here.