Lead Software Engineer - Big Data

About the role

The Lead Software Engineer is responsible for the development and maintenance of software products, ensuring high-quality standards and mentoring junior engineers.

Responsibilities

• Works on complex projects requiring extensive knowledge of specialty areas and industry practices
• Manages processes, projects, or products to enhance efficiency within the organization or product area
• Influences outcomes in cross-functional settings
• Designs and develops industry-leading solutions meeting defined quality standards and reliability specifications
• Ensures the quality and consistency of work across the team
• Implements methods for managing and maintaining the organization's software development environments
• Identifies and implements solutions to address gaps between system components
• Develops actionable insights from analyzing software development trends and communicates integration recommendations to management
• Collaborates with team members to integrate new technologies and methodologies
• Shares progress and priorities with key stakeholders, communicating complex or technical information clearly

Requirements

• Seasoned contributor with extensive knowledge of specialty areas and industry practices
• Proven ability to diagnose and resolve data skew, resource limitations, scalability challenges, and job failures in production environments
• Advanced proficiency in SQL, including window functions, multi-table joins, aggregations, and query optimization
• Strong experience with cloud technologies, including AWS services like S3, EMR, Glue, Lambda, Athena, EKS, and serverless architectures
• Expertise in big data technologies, including Hadoop, Spark, Hive, and Trino for processing petabyte-scale datasets
• Deep understanding of Spark architecture, including executors, tasks, stages, and DAG execution
• Strong experience with programming languages such as Python and Scala, with a strong understanding of functional programming concepts
• Hands-on experience with AI development tools and workflow design
• Extensive experience with agile methodologies and change management
• Strong system design experience with ability to architect scalable, distributed data processing solutions and enterprise-level applications
• Strong experience in object-oriented development, design patterns, and software engineering best practices
• Experience with modern data storage solutions, database technologies, and data modeling approaches for both relational and NoSQL systems
• Expertise in performance tuning and optimization, including identifying bottlenecks and optimizing application and data processing performance
• Strong experience with DevOps practices, continuous integration, continuous deployment pipelines, and infrastructure as code
• Strong understanding of security best practices, secure coding principles, and application security across all layers
• Ability to interpret AI-generated insights and data-driven metrics, translating them into actionable technical improvements and measurable business value

Qualifications

• Bachelor’s degree in Computer Science, Information Systems, or related field with at least 7 years of related experience, or equivalent training and/or work experience
• Strong system design experience
• Strong experience in object-oriented development
• Strong experience with cloud technologies
• Strong experience in data storage technologies
• Strong experience in performance tuning and optimization
• Strong experience in DevOps and CI/CD technologies
• Strong experience in test automation and unit testing
• Strong experience in software security