Sr. Data Engineer

Responsibilities

• Help design, develop, and optimize scalable data pipelines and data solutions using Databricks, PySpark, SQL, and Python.
• Help build and maintain reliable data models and ETL/ELT processes that support analytics, business intelligence, machine learning and AI.
• Help establish and maintain data engineering standards, governance, security, and best practices across the Azure and Databricks platforms.
• Evaluate new Databricks capabilities and recommend adoption strategies, standards, and implementation patterns.
• Monitor, troubleshoot, and optimize data pipelines and platform performance for reliability, scalability, and cost efficiency.
• Partner with cross-functional teams to solve complex data challenges and provide technical guidance on data architecture and engineering best practices.
• Mentor junior engineers and contribute to a culture of technical excellence and continuous improvement.

Qualifications

• 5+ years of experience in Data Engineering, Data Platform Engineering, or a related field.
• Strong hands-on experience with Databricks, Apache Spark, PySpark, Python, SQL, Azure Data Lake Storage, and Delta Lake.
• Deep understanding of distributed data processing and large-scale data architectures.
• Strong knowledge of data engineering principles, including data modeling, ETL and ELT design patterns, Data quality and observability, Metadata management, Data lifecycle management, Master Data Management concepts.
• Strong understanding of modern table formats and storage architecture.
• Experience with performance optimization techniques.
• Experience implementing CI/CD and infrastructure automation using tools.
• Excellent problem-solving, analytical, and communication skills.
• Ability to operate independently while leading initiatives in a fast-paced, agile environment.
• Passion for building scalable, maintainable, and well-governed data platforms.