Senior Data Developer Tech Lead - Vice President

Responsibilities

• Lead the design and implementation of scalable and efficient Hadoop architecture solutions, ensuring alignment with enterprise data strategy and best practices for small, co-located teams.
• Drive the design, development, and management of complex Hadoop-based architectures and real-time streaming data pipelines, demonstrating high autonomy and end-to-end ownership.
• Design and implement robust real-time data streaming solutions using technologies like Apache Kafka, proactively identifying and integrating performance enhancements.
• Collaborate effectively within a small, agile squad, translating diverse data requirements from engineers and scientists into detailed technical specifications and architectural blueprints.
• Optimize Hadoop clusters and real-time streaming applications for performance, scalability, and efficient resource utilization, ensuring high throughput and low latency.
• Maintain and monitor the Hadoop and streaming infrastructure to ensure high availability, reliability, and data integrity, proactively identifying and resolving issues with minimal oversight.
• Implement comprehensive data security and governance policies, adhering to strict regulatory and compliance standards prevalent in the banking sector, with a deep understanding of their functional impact.
• Stay abreast of the latest advancements and emerging trends in big data technologies, real-time analytics, AI/ML, and AI coding tools, recommending and integrating innovative, AI-first solutions to raise the team's capabilities.
• Troubleshoot and resolve complex issues within the Hadoop ecosystem and real-time streaming platforms efficiently, acting as a player/coach for less experienced team members.
• Develop Spark-based solutions to support near real-time data ingestion, advanced analytics, and comprehensive reporting, demonstrating hands-on expertise.
• Lead the contribution to the implementation of AI/ML models within data pipelines, focusing on data preparation, feature engineering, and seamless model deployment in production environments, and coach others on best practices.
• Drive MLOps initiatives to streamline the machine learning lifecycle, including continuous integration, continuous delivery, and continuous training, emphasizing efficiency gains through AI-driven development.
• Take full ownership of end-to-end project delivery for data initiatives, from conceptualization and architectural design to implementation, testing, and deployment, ensuring projects are delivered on time, within scope, and with exceptional quality.

Qualifications

• 10+ years of proven experience in designing, implementing, and managing complex Hadoop-based architectures and real-time data platforms, with a strong emphasis on senior, hands-on contributions.
• Demonstrated architectural leadership experience in big data environments, capable of operating as a player/coach.
• Strong understanding of Hadoop ecosystem components, including HDFS, YARN, MapReduce, Hive, HBase, and Spark.
• Extensive hands-on experience with real-time data streaming technologies, particularly Apache Kafka, and a track record of optimizing their performance.
• Demonstrable proficiency and experience using AI coding tools for accelerated and efficient development.
• Strong hands-on and architectural knowledge of Python, PySpark, Unix, and SQL.
• Familiarity with major cloud platforms such as AWS, Azure, and Google Cloud.
• Experience with advanced data modeling principles and practices, including dimensional modeling and data vault.
• Proficiency in designing and implementing complex ETL/ELT processes for both batch and real-time data.
• In-depth knowledge of data warehousing concepts and best practices.
• Strong understanding and practical experience with AI/ML lifecycle management, MLOps practices, and the integration of machine learning models into data solutions, with an "AI-first" mindset.
• Experience with Generative AI (GenAI) concepts and tools, and the ability to leverage them in practical applications, is a significant advantage.
• Proven track record of successful end-to-end project delivery for complex data initiatives.
• Excellent problem-solving skills and the ability to operate with high autonomy as part of a small, high-performing team.
• Strong communication skills, both written and verbal, with the ability to articulate complex technical concepts, mentor others, and raise the bar for the team.