Staff Data Engineer

About the role

The ideal candidate will design, develop, and maintain enterprise-scale data infrastructure leveraging Azure and Databricks technologies. This role involves building robust data pipelines, optimizing data workflows, and ensuring data quality and governance across the platform.

Responsibilities

• Design, build, and optimize data pipelines and workflows in Azure and Databricks, including Data Lake and SQL Database integrations.
• Implement scalable ETL/ELT frameworks using Azure Data Factory, Databricks, and Spark.
• Optimize data structures and queries for performance, reliability, and cost efficiency.
• Drive data quality and governance initiatives, including metadata management and validation frameworks.
• Collaborate with cross-functional teams to define and implement data models aligned with business and analytical requirements.
• Maintain clear documentation and enforce engineering best practices for reproducibility and maintainability.
• Ensure adherence to security, compliance, and data privacy standards.
• Mentor junior engineers and contribute to establishing engineering best practices.
• Support CI/CD pipeline development for data workflows using GitLab or Azure DevOps.
• Partner with data consumers to publish curated datasets into reporting tools such as Power BI.
• Stay current with advancements in Azure, Databricks, Delta Lake, and data architecture trends.

Requirements

• Advanced proficiency in Azure (5+ years).
• Strong expertise in Databricks (5+ years), Apache Spark (5+ years), and Delta Lake (5+ years).
• Proficient in SQL (10+ years) and Python (5+ years); familiarity with Scala is a plus.
• Strong understanding of data modeling, data governance, and metadata management.
• Knowledge of source control (Git), CI/CD, and modern DevOps practices.
• Familiarity with Power BI visualization tool.

Qualifications

• Bachelor’s or Master’s degree in Computer Science, Data Science, or related field.
• 7+ years of experience in data engineering, with significant hands-on work in cloud-based data platforms (Azure).
• Experience building real-time data pipelines and streaming frameworks.
• Strong analytical and problem-solving skills.
• Proven ability to lead projects and mentor engineers.
• Excellent communication and collaboration skills.

Preferred Qualifications

• Master’s degree in Computer Science, Engineering, or a related field.
• Exposure to machine learning integration within data engineering pipelines.