Databricks Engineer

Duration: 6-12 months
Location: 3 days onsite in Lexington, MA – local candidates only
Compensation: $60-75/hour (W2 preferred)

About the role

Join a fast-moving initiative supporting major data lake buildouts across commercial and technical operations in a life sciences environment. In this role, you’ll help keep mission-critical Databricks pipelines running smoothly—monitoring, troubleshooting, upgrading, and optimizing—while also jumping into development work as needed. You’ll collaborate closely with both technical teammates and non-technical stakeholders to deliver scalable, governed data products in a highly regulated setting.

Responsibilities

• Build and maintain data ingestion pipelines
• Monitor scheduled Databricks jobs and resolve production issues
• Troubleshoot, upgrade, and performance-tune Databricks pipelines
• Integrate and transition workflows from Informatica PowerCenter to IDMC
• Design and implement scalable, governed data solutions that meet regulatory requirements
• Translate business and technical requirements into robust data architectures
• Collaborate with cross-functional teams to align technical delivery with stakeholder needs

Requirements

• 8+ years of data/analytics engineering experience on modern data platforms
• Hands-on expertise with Databricks: PySpark, Delta Lake, pipeline troubleshooting, performance tuning, and upgrades
• Proven experience with Informatica PowerCenter and IDMC (including integrations and PowerCenter-to-IDMC transitions)
• Strong programming and query skills in Python and SQL
• Experience with Azure Data Factory
• Working knowledge of data lake architecture, data governance, data quality, and coding standards
• Ability to translate requirements into scalable solution designs
• Strong communication and stakeholder management skills
• Life sciences / pharma Databricks experience

Nice to Have

• Experience in commercial data domains (in addition to technical operations)
• Experience with MDM and enterprise governance practices
• Hands-on expertise implementing a data catalog (Unity Catalog) and creating data products
• Experience leveraging AI-enabled data engineering to optimize pipelines and data lake performance
• Consulting background with both technical and functional strengths
• Databricks and/or Informatica certifications

Qualifications

• 8+ years of relevant data engineering experience
• Experience in pharmaceutical or highly regulated environments (strongly preferred)

Pay

$60-75/hour (W2 preferred)

Schedule

3 days onsite in Lexington, MA