Lead Data Engineer

About the role

Lead design, development, and optimization of scalable data pipelines and ETL/ELT workflows using Azure Databricks, PySpark, SQL, and Delta Lake.

Responsibilities

• Lead design, development, and optimization of scalable data pipelines and ETL/ELT workflows using Azure Databricks, PySpark, SQL, and Delta Lake
• Enhancement of CI/CD pipelines and Azure DevOps, Git, and automated deployment frameworks to streamline development and release processes
• Ensure data quality and consistency through data integration, transformation, and cleansing
• Familiarity with security and governance best practices using Unity Catalog, Azure RBAC, managed identities, and data access controls to ensure compliance and secure data management is preferred
• Responsibility includes leading a team of data engineers and mentoring in best practices
• Utilize Python for data manipulation, automation, and custom data solutions
• Collaborate with stakeholders to understand data needs and translate them into technical solutions
• Participate in on-call rotation and assist with after-hours issues, as required
• Perform other duties as assigned

Requirements

• Bachelor's degree in computer science or information systems science, or equivalent experience
• Three (3) years experience with Azure Databricks platform
• Five (5) years of comprehensive data warehouse analysis and design experience, with full knowledge of data warehouse methodologies, data modeling and familiarity with operational data stores
• Five (5) years of experience with ETL/ELT Development, Python, PySpark
• Python, SQL, and Terraform experience

Qualifications

• Databricks Data Engineer Professional or Databricks Data Engineer Associate Certification preferred
• Ability to adapt to changing priorities and respond to problems creatively
• Ability to assist clients with migrating from legacy to modern data platform
• Proficiency with technical, conceptual, interpersonal, written, and verbal communication skills
• Knowledge of general business, quality and regulatory requirements preferred
• Familiarity with Power BI and MicroStrategy architecture is a plus

Physical Requirements

• Position works in-office at the corporate location
• Ability to sit for extended periods of time
• Ability to use a computer and other office equipment
• Ability to occasionally lift and carry up to 20 pounds
• Ability to reach, bend, and stoop as necessary
• Ability to communicate effectively, both verbally and in writing
• Ability to focus and concentrate on tasks for extended periods
• Ability to navigate the office environment safely, including stairs and elevators (if applicable)
• Ability to travel up to 20% via airplane or vehicle (if applicable)
• Ability to use assistive devices if needed for mobility or communication

Benefits

• Formal training
• Outstanding plans for medical, dental, and vision insurance
• Health savings account (HSA)
• Flexible spending account (FSA)
• Tuition Reimbursement
• Employee assistance program (EAP)
• Wellness program
• 401k retirement plan
• Paid time off
• Company paid holidays
• Personal time