Software Development Engineer, Open Data Analytics - Engines

About the role

Amazon Web Services Open Data Analytics (ODA) organization is seeking an experienced engineer to join the core engines and datalake team. This role focuses on enhancing AWS services such as EMR and Athena, which enable customers to run large-scale analytics using open-source engines like Apache Spark and Trino, alongside open table formats like Apache Iceberg, Hudi, and Delta.

The team makes significant modifications to these engines to optimize performance, scalability, and serverless execution, achieving up to 5x improvements in optimizer, query runtime, and storage connectors over the past three years. You will contribute to compiler enhancements for enterprise features like fine-grained access control and regularly contribute innovations back to open-source communities.

This is a strategic, high-impact role in a growing technical space, where you will collaborate with open-source communities, drive performance optimizations, and shape the technical direction of query engines and table format integrations.

Responsibilities

• Develop and optimize core components of query engines and open table formats (Iceberg, Hudi, Delta) to enhance performance, scalability, and reliability.
• Design and implement innovative solutions and algorithms to improve feature capabilities, stability, and security in table format integrations with query engines.
• Collaborate with the open-source community, contributing to discussions, driving improvements, and integrating upstream changes.
• Ensure data consistency and durability while achieving breakthrough performance and scalability for large-scale data lake workloads.
• Improve the organization’s automation and testing capabilities.
• Manage complex deliverables, projects, and research with deadlines.
• Mentor and train other team members on design techniques and coding best practices.
• Serve as a point of contact for challenging customer issues related to data lake workloads and query engines.

Requirements

• 3+ years of non-internship professional software development experience.
• 2+ years of non-internship design or architecture experience (design patterns, reliability, and scaling) of new and existing systems.
• 1+ years of experience designing and developing large-scale, multi-tiered, multi-threaded, embedded, or distributed software applications, tools, systems, and services using C#, C++, Java, or Perl.
• 1+ years of Object-Oriented Design experience.
• Bachelor’s degree or foreign equivalent in Computer Science, Engineering, Mathematics, or a related field.
• Experience programming with at least one software programming language.

Preferred Qualifications

• 3+ years of full software development life cycle experience, including coding standards, code reviews, source control management, build processes, testing, and operations.
• Bachelor’s degree in Computer Science or equivalent.

Benefits

• Comprehensive health insurance (medical, dental, vision, prescription, Basic Life & AD&D, and optional supplemental life plans).
• Employee Assistance Program (EAP) and mental health support.
• Medical Advice Line and Flexible Spending Accounts.
• Adoption and surrogacy reimbursement coverage.
• 401(k) matching.
• Paid time off and parental leave.
• Sign-on payments and restricted stock units (RSUs).

Pay

Base salary ranges:

• East Palo Alto, CA: $165,200 - $223,600 USD annually.
• Redmond, WA: $143,700 - $194,400 USD annually.

Final compensation will be determined based on experience, qualifications, and location.

About the team

AWS Utility Computing (UC) drives product innovations, from foundational services like Amazon S3 and EC2 to new advancements in compute, database, storage, IoT, platform, and productivity apps. As part of this organization, you’ll support the development and management of AWS services, including specialized security solutions for customers.

The team values collaboration across diverse projects and environments, solving challenging technical problems at every layer of the stack. Engineers design, implement, test, deploy, and maintain innovative software solutions to transform service performance, durability, cost, and security.