Lead Data Engineer

Responsibilities

• Lead the design and implementation of scalable big data pipelines for batch and real time processing
• Build and operate streaming data platforms using Kafka
• Design and deploy cloud native data solutions across AWS, Azure, and GCP
• Develop and manage containerized workloads using Kubernetes
• Enable data mesh architecture with domain oriented data products
• Design and implement data lake and lakehouse architectures
• Ensure data quality, reliability, and observability
• Implement governance capabilities including metadata and lineage
• Collaborate with analytics, AI, and business teams
• Optimize performance and cost efficiency
• Mentor engineering teams

Requirements

• 15+ years of experience in big data engineering
• Strong experience with Kafka
• Strong experience with Kubernetes
• Experience with AWS, Azure, and GCP
• Experience building data lakes or lakehouse platforms
• Experience with data mesh concepts
• Strong programming skills in Python, Scala, or Java
• Experience with Spark, Flink, or Beam
• Experience with Airflow or orchestration tools
• Understanding of data modeling and governance
• Experience with CICD and infrastructure automation
• Experience supporting AI and machine learning workloads including large language models such as Claude or similar platforms
• Strong communication and leadership skills

Qualifications

NorthMark Compute & Cloud (NMC²) is committed to diversity, equity, and inclusion. We welcome applications from all qualified individuals.