Senior Azure Databricks Data Engineer

About the role

This Sr. Engineer 2 role is a hands-on, senior-level engineering position focused on designing and implementing scalable data and application solutions using Python, Spark, Databricks, and Azure Cloud (including Azure ML). The position is hybrid, requiring onsite work Tuesday through Thursday and virtual work on Mondays and Fridays. The engineer will work across multiple agile pods, driving best practices in data engineering, DevOps, monitoring, and governance, while collaborating closely with data scientists, analysts, and business stakeholders to deliver robust, production-grade solutions.

Responsibilities

• Participate in and contribute to end-to-end architecture, design, and implementation of scalable data and application solutions using Python and Azure ML.
• Design and develop complex ETL/ELT pipelines to process, transform, and integrate large-scale datasets from diverse sources using Spark and Python.
• Optimize Python and Spark jobs for performance and cost-efficiency through techniques such as partitioning, caching, cluster configuration tuning, and bottleneck analysis.
• Collaborate with data scientists, analysts, and business stakeholders to translate complex business problems into robust, maintainable technical solutions.
• Implement and enforce data governance and security strategies, including access controls, encryption, data lineage, and frameworks such as Unity Catalog.
• Champion code quality, engineering best practices, and standards across pods, including code reviews, testing, and documentation.
• Design, build, and maintain DevOps CI/CD pipelines (e.g., Azure DevOps, GitHub Actions, Jenkins) to automate build, test, deployment, and management of data assets and infrastructure.
• Develop and implement comprehensive monitoring, logging, and alerting systems for production-grade pipelines to proactively detect, troubleshoot, and resolve issues.
• Support multiple agile pods by guiding project planning, sprint execution, and tracking deliverables across various initiatives.
• Promote and help enforce agile methodologies across pods, ensuring consistent ceremonies, backlog management, and delivery tracking.
• Leverage AI tools and automation to improve productivity, code quality, and operational efficiency.
• Participate in technical interviews, including coding assessments, and contribute to the evaluation of candidates when needed.
• Work effectively in a hybrid environment, attending mandatory onsite meetings and interviews as required.

Qualifications

• 7–10 years of overall experience in software engineering and data engineering roles.
• At least 7 years of hands-on experience with Python, ETL development, data handling, and SQL.
• Advanced proficiency in Python, Scala, and/or SQL for data processing and application development.
• Minimum 5 years of experience with Spark and Databricks programming in production environments.
• Strong experience designing and implementing complex ETL/ELT pipelines for large-scale datasets.
• Solid background in Azure Cloud services, including Azure ML and other relevant Azure data and application services.
• Experience implementing data governance, security, and compliance frameworks, including access control, encryption, and data lineage (e.g., Unity Catalog).
• Proven experience designing and developing CI/CD pipelines that deploy code across multiple development and production environments, preferably on Azure using GitHub Actions or similar tools.
• At least 5 years of experience with logging, monitoring, and alerting systems for production workloads.
• Strong understanding of the full software development life cycle (SDLC), including requirements, design, development, testing, deployment, and support.
• Experience using AI tools to enhance productivity, code quality, or data workflows.

Nice to have

• Experience with Java and Spring for application development.
• Broad knowledge of different Azure services and how they integrate within data and application architectures.
• Excellent communication skills, with the ability to explain complex technical concepts to both technical and non-technical stakeholders.
• Demonstrated ability to work in and support agile, pod-based team structures, including cross-team coordination and planning.
• Ability to work in a hybrid schedule, with mandatory onsite presence Tuesday through Thursday and flexibility to work virtually on Mondays and Fridays.
• Comfort participating in a multi-round interview process, including a virtual interview, mandatory onsite interview, and coding assignment.

Benefits

• Medical, dental