Senior Data Engineer

About The Role

We are seeking a highly skilled and experienced Senior Data Engineer to join our growing data and machine learning organization and help build the pipelines, models, and infrastructure that power our analytics, machine learning, and operational data needs.

In this role, you will work closely with analysts, data scientists, ML/AI engineers, and product teams to design and deliver reliable, scalable data workflows on our Databricks Lakehouse platform.

• Data Pipeline Engineering
  • Design, build, and maintain scalable ETL/ELT pipelines using Spark, Python, SQL, and Databricks.
    • Implement reliable ingestion frameworks for batch and streaming data sources.
    • Ensure pipelines meet SLAs, data quality standards, and production-grade reliability.
• Lakehouse Modeling & Architecture
  • Create robust data models across raw, curated, and semantic layers using Delta Lake.
  • Establish and maintain standards for schema design, metadata, and lineage.

Data Quality & Observability

• Implement data validation, anomaly detection, SLAs, and documentation across pipelines.

• Build automated tests, monitoring, and alerting for freshness, completeness, and accuracy.

• Partner with platform teams to enhance observability and operational tooling.

Collaboration & Cross-Functional Support

• Work closely with analysts to understand business KPIs and deliver high-quality curated datasets.

• Partner with ML engineers and data scientists to build reusable feature pipelines.

• Collaborate with data platform engineers to optimize compute, governance, and orchestration.

Performance & Optimization

• Optimize Spark jobs, SQL queries, cluster configurations, and storage patterns for performance and cost.

• Improve reliability, reduce technical debt, and simplify complex pipelines.

Security, Compliance & Governance

• Apply best practices for RBAC, data privacy, and PII handling using Unity Catalog.

• Ensure adherence to compliance frameworks and documentation standards.

Continuous Learning

• Stay current on modern data engineering patterns, Lakehouse architecture, orchestration, and best practices.

• Explore new technologies that improve reliability, scalability, and developer productivity.

What You’ll Do

Data Pipeline Engineering

Design, build, and maintain scalable ETL/ELT pipelines using Spark, Python, SQL, and Databricks.

Implement reliable ingestion frameworks for batch and streaming data sources.

Ensure pipelines meet SLAs, data quality standards, and production-grade reliability.

Lakehouse Modeling & Architecture

Develop robust data models across raw, curated, and semantic layers using Delta Lake.

Create dimensional models, star schemas, and domain-layer datasets for analytics and ML.

Establish and maintain standards for schema design, metadata, and lineage.

Data Quality & Observability

Implement data validation, anomaly detection, SLAs, and documentation across pipelines.

Build automated tests, monitoring, and alerting for freshness, completeness, and accuracy.

Partner with platform teams to enhance observability and operational tooling.

Collaboration & Cross-Functional Support

Work closely with analysts to understand business KPIs and deliver high-quality curated datasets.

Partner with ML engineers and data scientists to build reusable feature pipelines.

Collaborate with data platform engineers to optimize compute, governance, and orchestration.

Performance & Optimization

Optimize Spark jobs, SQL queries, cluster configurations, and storage patterns for performance and cost.

Improve reliability, reduce technical debt, and simplify complex pipelines.

Security, Compliance & Governance

Apply best practices for RBAC, data privacy, and PII handling using Unity Catalog.

Ensure adherence to compliance frameworks and documentation standards.

Continuous Learning

Stay current on modern data engineering patterns, Lakehouse architecture, orchestration, and best practices.

Explore new technologies that improve reliability, scalability, and developer productivity.