Senior Data Engineer

Responsibilities

• Own the design and delivery of complex data engineering solutions that power NYBCe's enterprise analytics, AI, and reporting capabilities.
• Drive technical decisions, set engineering standards, and ensure the reliability and scalability of DAPI's data platform across 49+ integrated enterprise source systems.
• Architect, build, and own complex data pipelines for high-volume, high criticality workstreams across NYBCe's enterprise data platform.
• Lead the design and implementation of ELT/ETL frameworks using SQL, Python, Azure Data Factory, Databricks, and Azure Synapse Analytics.
• Establish pipeline reliability standards—monitoring, alerting, error handling, and recovery protocols—and ensure adherence across the team.
• Drive the design of scalable data models supporting dimensional warehousing, data lake architectures on Azure.
• Contribute to architectural decisions on data storage, partitioning, compute optimization, and consumption layer design.
• Lead migrations from legacy data solutions to modern cloud-native platforms, managing risk and business continuity throughout.
• Design and deliver feature pipelines and data preparation frameworks that support machine learning model development and deployment.
• Partner with Data Scientists to translate model requirements into production-grade data assets and feature stores.
• Collaborate with Analytics Engineers to ensure data models are optimized for analytical consumption and reporting performance.
• Define and implement data quality frameworks—validation rules, SLAs, anomaly detection, and automated testing for pipeline outputs.
• Lead data governance initiatives including metadata management, lineage tracking, data cataloging (Microsoft Purview), and access control.
• Ensure platform compliance with HIPAA, NYBCe data policies, and applicable regulatory requirements.
• Mentor Data Engineers—providing code reviews, technical guidance, and architectural feedback that elevates team capability.
• Contribute to DAPI's engineering standards, reusable frameworks, and technical documentation.
• Participate in Agile ceremonies and model strong engineering discipline—clear DevOps hygiene, sprint commitment, and delivery accountability.

Qualifications

• Bachelor’s degree in computer science, Data Science, Information Technology, or a related quantitative field.
• 6+ years of progressive experience in data engineering with demonstrated ownership of complex, production-grade data platforms.
• Expert-level SQL (query optimization, indexing strategy, execution plans) and Python (PySpark, pipeline frameworks, testing).
• Deep hands-on experience with Azure data services: Azure Data Factory, Azure Databricks, Azure Synapse Analytics, Azure Data Lake Storage.
• Proven experience designing dimensional data models and data lake architecture at enterprise scale.
• Experience building data pipelines that directly support machine learning feature engineering and model serving.
• Strong background in data quality engineering—automated validation, SLA enforcement, and lineage tracking.
• Experience with relational databases (SQL Server, Oracle) and migration from legacy to cloud-native platforms.
• No certifications are required. The following are considered favorable: Microsoft Certified: Azure Data Engineer Associate, Databricks Certified Associate Developer for Apache Spark.
• Knowledge: Advanced SQL and Python for enterprise-scale data engineering—optimization, testing, and framework design, Azure data platform architecture in depth—ADF, Databricks, Synapse, ADLS, and their integration patterns, Modern data platform paradigms—data lake, medallion architecture, data mesh concepts, and consumption layer design, Machine learning pipeline requirements—feature engineering, training data preparation, and model data dependencies, Data governance frameworks—metadata management, lineage, cataloging, access control, and regulatory compliance (HIPAA), Agile engineering practices—sprint delivery, DevOps hygiene, CI/CD for data pipelines, and technical documentation standards.
• Skills: Architect and deliver complex, production-grade data pipelines that meet enterprise reliability and performance standards, Design scalable data models and platform structures that serve analytics, reporting, and AI consumption patterns simultaneously, Lead data quality engineering—automated testing, validation frameworks, SLA monitoring, and incident response, Mentor and elevate Data Engineers through code review, architectural feedback, and knowledge transfer, Translate product and analytical requirements into sound engineering designs and delivery plans.
• Abilities: Communicate complex data concepts clearly to both technical and non-technical stakeholders, Work independently and manage competing priorities in a lean, fast-paced team environment, Embrace accountability and take ownership of deliverables end-to-end, Incorporate feedback constructively and seek continuous improvement.