Lead Data Platform Engineer

Lead Data Platform Engineer - Databricks / Delta Lake / Unity Catalog - $220-$250k DOE + Benefits - San Diego, CA & Tarrytown, NY (Hybrid)

Location: San Diego, CA & Tarrytown, NY- Hybrid (in-office 2-3 days/week)
Salary: $220k-$250k base DOE
Benefits: Comprehensive health, 401(k), generous PTO, parental leave, supplemental benefits

What's in it for you?

• Lead the architecture and build of a centralized, multi-product data platform from the ground up on Databricks.
• Solve challenging data infrastructure problems at scale-multi-TB datasets, Delta Lake optimization, and cross-product governance.
• Hands-on role enabling other Data Engineers while defining platform operations, performance, and reliability standards.
• Exposure to cutting-edge technologies: Unity Catalog, Delta Live Tables, PySpark, Databricks SQL, AWS, Terraform.
• Be part of a collaborative, growth-focused team with opportunities to influence technical strategy and career development.

The Role

As Lead Data Platform Engineer, you will:

• Design and implement unified data models across multiple disparate product lines.
• Build a multi-catalog governance strategy with Unity Catalog, enabling secure isolation and controlled cross-product data sharing.
• Optimize Delta Lake tables with Z-ordering, compaction, liquid clustering, partitioning, and retention policies.
• Create declarative ETL pipelines with Delta Live Tables, orchestrating data ingestion from internal and external sources.
• Integrate third-party data sources (ERP systems, external providers) with automated, monitored ingestion.
• Establish platform operations standards: cost monitoring, performance tuning, data quality frameworks, and self-service capabilities for engineers.

Business Challenges You'll Solve

• Lead migration of multi-terabyte datasets from legacy systems to a unified Databricks lakehouse.
• Design multi-product data architectures enabling both secure isolation and analytics sharing where appropriate.
• Build a platform that scales efficiently, controlling costs while supporting analytics and AI workloads.
• Implement monitoring, alerting, and validation to ensure reliability and high-quality data.

Required Experience

Databricks Expertise

• Unity Catalog: production experience with multi-catalog governance, metastore design, lineage tracking
• Delta Lake: Z-ordering, compaction, liquid clustering, performance tuning at multi-TB scale
• Delta Live Tables: declarative ETL pipelines, change data capture, expectations/constraints
• Databricks Workflows: job orchestration, scheduling, monitoring
• PySpark & Databricks SQL: query optimization, code review, performance tuning

Core Platform Engineering

• 6-8 years in data engineering/platform roles, with 3+ years hands-on Databricks experience
• Led at least one major platform build or migration
• AWS experience (S3, IAM, VPC), Infrastructure-as-Code (Terraform preferred)

Technical Leadership

• Architect platforms from first principles and communicate technical decisions clearly
• Strong written and verbal communication for technical and business stakeholders

Preferred (Not Required)

• Experience with financial/ERP data (NetSuite)
• Exposure to AI/ML data preparation or RAG/LLM pipelines
• Familiarity with regulated industry compliance and governance frameworks

Why This Role?

• Take ownership of mission-critical data infrastructure impacting multiple business lines.
• Work on complex, large-scale datasets in a cutting-edge data platform environment.
• Opportunity to mentor and enable a team of Data Engineers while shaping long-term architecture.
• Competitive salary, bonus, benefits, and career growth in a collaborative, hybrid environment.

Apply Now - This role is exclusive to our recruiting partner, and limited interview slots are available. Contact Bryce Reading for more information or to submit your application.

Lead Data Platform Engineer / Databricks / Delta Lake / Unity Catalog / Delta Live Tables / Data Governance / PySpark / Data Engineering / AWS / Terraform / San Diego / NYC / Python

Oscar Associates Limited (US) is acting as an Employment Agency in relation to this vacancy.