Lead Data Platform Engineer

Oscar is working with a leading AI FinTech SaaS start-up organization that is looking for an experienced Lead Data Platform Engineer to join their team.

As the Lead Data Platform Engineer, you will design, build, and scale our centralized data platform on Databricks. You will define platform architecture, establish governance standards using Unity Catalog, and optimize the platform for performance, reliability, and cost efficiency at scale.

This role focuses on data infrastructure and platform engineering-including pipelines, storage architecture, governance, and operational excellence. You will also enable our Data Engineering team to build confidently on top of a well-governed, scalable platform.

Key Responsibilities:

• Architect and implement our Databricks lakehouse platform

• Design governance patterns using Unity Catalog

• Build scalable Delta Lake data models and optimization strategies

• Develop ETL frameworks using Delta Live Tables

• Integrate internal systems (SQL databases, S3) and external providers (ERP systems, financial data sources)

• Lead migration of multi-terabyte datasets from legacy platforms

• Implement monitoring, data quality, and cost optimization practices

• Enable self-service data engineering with strong governance standards

Required Qualifications:

• 8+ years in data engineering or data platform roles

• 3+ years hands-on Databricks experience

• Production experience with:

  • Unity Catalog

  • Delta Lake optimization (Z-ordering, clustering, compaction)

  • Delta Live Tables

  • Databricks Workflows

  • PySpark and Databricks SQL

• Experience leading a major platform build or large data migration

• AWS experience (S3, IAM, VPC)

• Start up background/experience

• Infrastructure-as-code experience (Terraform preferred)

Recap:

Location: Tarrytown, NY (Onsite)

Type: Permanent Fulltime

Salary: 220k-250k + Benefits

If you think you're a good fit for the role, we'd love to hear from you!

Oscar Associates Limited (US) is acting as an Employment Agency in relation to this vacancy.