Lead Data Platform Engineer

About the Role

A growing, multi-product enterprise organization in the financial data space is seeking a Lead Data Platform Engineer to architect and scale its centralized Databricks lakehouse. This is a highly strategic, infrastructure-focused role responsible for designing and implementing the data platform that powers analytics and AI across multiple product lines. The organization is actively modernizing its ecosystem, migrating multi-terabyte legacy datasets into Databricks while establishing scalable governance, performance optimization, and operational best practices.

*Unfortunately this role does not offer sponsorship. U.S. Citizens/Greencard Holders Only*

Details:

• Full-Time, Permanent Position
• $180k - $220k + Benefits
• San Diego, CA | Hybrid - 3 Days On-Site

Key Responsibilities

• Architect and implement a centralized Databricks lakehouse
• Design unified data models across multiple product lines
• Optimize Delta Lake tables (Z-ordering, compaction, clustering) at multi-terabyte scale
• Establish Unity Catalog governance and multi-catalog access strategies
• Develop declarative ETL pipelines using Delta Live Tables
• Lead large-scale legacy data migrations into a unified platform
• Implement cost monitoring, data quality frameworks, and platform reliability standards

Required Qualifications

• 8+ years of experience in data engineering or data platform roles
• 3+ years of hands-on Databricks experience
• Strong expertise in Unity Catalog, Delta Lake optimization, and Delta Live Tables
• Proficiency in PySpark and Databricks SQL
• Experience leading a major data platform build or migration initiative
• AWS experience (S3, IAM, VPC)
• Infrastructure-as-Code experience (Terraform preferred)

Oscar Associates Limited (US) is acting as an Employment Agency in relation to this vacancy.