Azure Databricks Engineer

About the role

As a Databricks Support Specialist for the Pharma industry, you will be responsible for ensuring the operational stability and performance of Databricks-based data platforms. Your expertise in ITSM processes and monitoring tools will drive efficient incident resolution and proactive problem management, supporting critical data analytics operations for life sciences clients.

Responsibilities

• Provide L2/L3 support for Databricks environments, ensuring high availability and performance of data pipelines.
• Manage incidents, service requests, and changes using ServiceNow, adhering to ITIL frameworks.
• Monitor Databricks clusters and jobs using Dynatrace and BigPanda, identifying and resolving performance bottlenecks.
• Troubleshoot ETL workflow failures, data quality issues, and integration problems with clinical and operational datasets.
• Collaborate with data engineering teams to escalate complex issues and coordinate minor enhancements.
• Document known issues, troubleshooting guides, and runbooks in Confluence.
• Track support activities in JIRA, ensuring SLA adherence and quality reporting.
• Perform routine maintenance and automation tasks using shell scripts and basic Python/SQL.
• Coordinate with cloud infrastructure teams for AWS/Azure-related issues affecting Databricks environments.
• Participate in on-call support rotations and incident bridges.

Requirements

• Bachelor's or Master's degree in Computer Science, Information Technology, Data Science, or a related field.
• 2+ years of experience in data platform support or IT operations, preferably with Databricks or Spark.
• Strong understanding of ITSM processes and experience with ServiceNow for incident and change management.
• Proficiency with Dynatrace, BigPanda, or similar monitoring and alerting tools.
• Experience with JIRA for ticket management and Confluence for knowledge documentation.
• Basic shell scripting and Python/SQL skills for troubleshooting and automation.
• Foundational knowledge of AWS and Azure cloud platforms.
• Basic understanding of ETL processes and data engineering concepts is a plus.
• Excellent analytical, problem-solving, and communication skills.
• Exposure to GenAI technologies and Gemini for AI-assisted operations and automation is a plus.