Lead Data Engineer (AWS)

Overview Being good neighbors – helping people, investing in our communities, and making the world a better place – is who we are at State Farm. It is at the core of how we operate and the reason for our success. Come join a #1 team and do some good! HYBRID: Qualified candidates must live or relocate within a 180-mile radius of a hub location listed below and should plan to spend time working from home and some time working in the office as part of our hybrid work environment. HUB LOCATIONS: Bloomington, IL ; Richardson, TX ; Tempe, AZ ; Dunwoody, GA SPONSORSHIP: Applicants for this position are required to be eligible to lawfully work in the U.S. immediately; employer will not sponsor applicants for U.S. work authorization (e.g. H-1B visa) for this opportunity. Responsibilities Innovation Group is seeking a Data Engineer or Lead Data Engineer to join our product execution team within SAIL (State Farm Applied Innovation Lab) focused on Agency and Digital. You will work alongside software engineers, product teams, and business partners to design and build the data capabilities that enable experimentation, product development, and emerging technology solutions. Our teams operate in environments where the problem, data, and solution may not be fully defined at the outset. We are looking for engineers who are comfortable working through that ambiguity, quickly learning new technologies and domains, and turning complex data needs into practical, scalable solutions. Day to day, the work centers on three areas: writing code to move and transform data; partnering with software engineers and product teams to determine how data can be accessed, analyzed, and used; and designing and maintaining the platforms where structured and unstructured data lives. Strong engineering fundamentals are essential, but success in this role also requires flexibility, curiosity, innovative thinking, and the ability to navigate a complex organization. The ideal candidate understands the fundamentals of data engineering and how data solutions are designed, built, deployed, monitored, and maintained as production-quality applications rather than one-off scripts. They communicate clearly, collaborate effectively across technical and non-technical teams, seek to understand the business problem behind the request, and continuously develop their skills as technologies and needs evolve. Depending on the initiative, this role may also help lead data science, machine learning, and generative AI efforts, working across the lifecycle from data exploration and experimentation through engineering, deployment, and ongoing operation. At the Lead Data Engineer level, the role also includes providing technical leadership, mentoring engineers, influencing architecture and engineering practices, and helping shape the direction of the team's data capabilities and solutions. Qualifications 3 plus years with python development for extracting, transforming, processing, and moving dataProficiency writing SQL queries and working with relational databasesExperience designing, building, and maintaining ETL/ELT data pipelinesExperience working with structured, semi-structured, and unstructured data, including formats such as CSV, JSON, and textExperience with public cloud platforms, preferably AWS, including services such as S3, Glue, and LambdaExperience using Git and modern version-control practicesFamiliarity with CI/CD practices and deploying code into shared or production environmentsComfort working in Linux and command-line environmentsUnderstanding of software engineering principles such as testing, maintainability, observability, and reusable designAbility to troubleshoot data quality, pipeline, integration, and performance issues Additional Skills That Would Be Valuable Experience designing data architectures for new products, experiments, or emerging technology use casesFamiliarity with machine learning concepts and the end-to-end ML lifecycle, including data preparation, feature engineering, model development, evaluation, deployment, monitoring, and iterationExperience partnering with data scientists or ML engineers to operationalize models and build the data pipelines and infrastructure that support machine learning solutionsFamiliarity with generative AI, large language models (LLMs), and the data considerations associated with building AI-enabled applicationsExperience designing or building LLM-based workflows, including agentic workflows that use tools, APIs, retrieval, orchestration, or multi-step reasoning to complete tasksExperience using AI-assisted development tools as part of the engineering workflow to accelerate coding, testing, debugging, documentation, and technical explorationAbility to evaluate AI-generated code and outputs critically and apply appropriate engineering, security, testing, and governance practicesExperience with APIs, event-driven architectures, or streaming dataFamiliarity with object storage and modern data platformsExperience working with large or complex datasetsUnderstanding of data governance, security, privacy, lineage, and access-management considerationsExperience working in Agile or product-oriented engineering environmentsAbility to translate business needs into technical data requirements and solutionsExperience working across teams and managing technical dependenciesComfort experimenting with new technologies and solving problems where the path forward is not yet fully defined