Manager, Data Engineering
Team Leadership & Delivery
Lead, coach, and develop a team of Data Engineers, fostering strong technical skills and ownership.
Set clear priorities, manage workload, and ensure timely delivery of data engineering initiatives.
Establish engineering standards, code quality expectations, and best practices across the team.
Partner with stakeholders to translate business needs into scalable data solutions.
Data Pipeline Engineering
Oversee the design, development, and operation of data pipelines built in Databricks.
Pipelines must be scalable, reliable, and aligned to medallion architecture standards (bronze, silver, gold).
Guide implementation of ingestion frameworks using tools like Fivetran and custom ingestion patterns.
Drive consistent development patterns for batch and near real-time data pipelines.
Platform & Architecture
Provide technical leadership for the Databricks platform, including workspace, jobs, clusters, and performance optimization.
Collaborate with Cloud Engineering to ensure seamless integration with AWS services (e.g., S3).
Ensure alignment with enterprise architecture, including data modeling, partitioning strategies, and storage optimization.
Optimize compute usage for performance and cost efficiency.
Data Quality, Governance & Reliability
Establish and enforce data quality standards, including testing, validation, and monitoring frameworks.
Ensure robust observability across pipelines (monitoring, alerting, lineage visibility).
Partner with Governance teams to support metadata management and lineage through tools like Atlan.
Enforce security, compliance, and data access standards across all data engineering assets.
Operations & Continuous Improvement
Own production support processes, including incident management, root cause analysis, and prevention.
Establish proactive monitoring and health checks for pipelines and platform performance.
Drive continuous improvement in automation, CI/CD, and release management practices.
Support and lead data migration efforts from legacy on-prem systems to cloud platforms.