Data Engineer

About the Role

This is a hybrid, office-based position working at our Ellis Preserve office located in Newtown Square, PA. Core office days are Tuesday, Wednesday, and Thursday. The Azure Data Engineer designs, builds, and optimizes large-scale cloud data solutions on Microsoft Azure, requiring deep hands-on expertise with modern ingestion frameworks, distributed compute, real-time streaming, and dimensional modeling, combined with agile delivery practices.

Responsibilities

• Architecture & Implementation: Design and implement scalable engineering solutions using Azure Data Factory, Azure Databricks, Azure Event Hubs, Kafka, HVR, Snowflake, and Azure Data Lake Storage.
• Pipeline Development: Build robust ETL/ELT pipelines for batch and streaming workloads using PySpark and SQL, ensuring fault tolerance, schema evolution handling, and idempotency.
• Real-time Integration: Design and operate event-driven ingestion and processing using Event Hubs and Kafka, and integrate CDC solutions using HVR.
• Performance Optimization: Tune Databricks jobs and clusters (cluster sizing, autoscaling, caching, partitioning, shuffle optimization, Delta Lake optimizations) to meet SLAs and cost targets.
• DevOps & CI/CD: Use Azure DevOps for backlog management, Git workflows, YAML pipelines, automated testing, and repeatable deployments across environments.
• Code Quality & Mentorship: Perform code reviews, enforce coding standards and best practices, and mentor junior developers to raise team capability.
• Stakeholder Collaboration: Work with product owners, data scientists, analysts, and architects to translate business requirements into prioritized user stories and deliverables in Agile sprints.
• Process Improvements: Identify, design, and implement internal process improvements, such as automating manual processes, optimizing data delivery, and re-designing infrastructure for greater scalability.
• Data Management: Identify and tackle issues concerning data management to improve data quality, including master data, reference data, metadata, data quality, and lineage.

Requirements

• Bachelor’s degree in Computer Science, Information Systems, or Software Engineering.
• Hands-on experience with:
  • Azure Data Factory (4+ years)
  • Azure Databricks (4+ years)
  • HVR (2+ years)
  • Kafka (4+ years)
  • Event Hubs (2+ years)
  • Snowflake (4+ years)
  • Azure Data Lake Storage
  • PySpark (8+ years)
  • SQL (8+ years)
• Expertise in dimensional modeling, SCD implementation, and data warehouse best practices.
• Proven experience delivering data projects in Agile (Scrum or Kanban) teams.
• Practical experience with Azure DevOps for CI/CD, branching strategies, and pipeline automation.
• Demonstrated ability to profile and optimize Databricks jobs and large-scale data processing workloads.
• Track record of conducting code reviews and providing technical guidance.
• Clear communication skills to present technical tradeoffs to both technical and non-technical stakeholders.

Preferred Qualifications

• Experience with Collibra Data Governance features.
• Familiarity with MLOps patterns and integrating data platforms with ML pipelines.
• Azure certifications such as DP-700, AZ-400, or Databricks Data Engineer certification.
• Prior work with cost optimization and observability tools for cloud data platforms.
• Practical experience applying Gen AI (LLMs, embeddings, retrieval augmentation) to data engineering workflows.

Skills

• Problem Solving: Analytical approach to diagnose performance and data quality issues.
• Collaboration: Strong team player who partners effectively across product, analytics, and platform teams.
• Mentorship: Patient, constructive coach who elevates code quality and engineering practices.
• Adaptability: Comfortable with shifting priorities and iterative delivery in fast-moving environments.

Benefits

• Flexible work solutions including hybrid work schedules.
• Competitive pay.
• Paid time off including holidays and volunteer events.
• Health insurance coverage for you and your dependents on Day 1.
• 401(k) retirement plan.
• Tuition reimbursement.