Sr. Software Engineer

Responsibilities

• Own and improve service reliability, availability, and performance across a distributed AWS platform (EKS, EC2, RDS, S3)
• Define and track SLOs, SLIs, and error budgets for critical services and partner-facing integrations
• Drive initiatives to identify manual, repetitive operational work and eliminate it through automation
• Partner with engineering teams to embed reliability practices into the SDLC
• Platform & Observability Engineering
• Build and maintain observability-as-code infrastructure using Terraform and New Relic NerdGraph
• Design multi-signal observability pipelines metrics, logs, traces across Splunk, New Relic, and OpenSearch
• Build New Relic NerdGraph integrations and custom NRQL dashboards for platform health visibility
• Maintain and extend Terraform modules for AWS infrastructure provisioning
• Contribute to the MCP server architecture (Python-based internal developer tooling) integrating Jira, Confluence, Bitbucket, and observability platforms
• Python Development
• Design, build, and maintain production-grade Python services and automation REST APIs, background workers, CLI tooling, and data pipeline adapters
• Mentor junior and mid-level engineers, code reviews, pair programming, architecture guidance
• Work closely with program managers and product stakeholders

Qualifications

• Bachelor's degree in Information Technology, Computer Science, or related field; or equivalent experience
• 7+ years of software engineering experience
• 4+ years of SRE or platform engineering experience in a production environment
• Strong Python proficiency production services, automation, REST APIs
• Experience with AWS EKS, EC2, RDS/Aurora, S3, IAM, VPC, CloudWatch, Security Groups
• Hands-on Terraform experience writing modules, managing state, CI/CD integration
• Experience with observability platforms Splunk New Relic (NRQL, NerdGraph)
• Strong understanding of SRE principles SLOs/SLIs/error budgets, toil reduction, incident management, capacity planning
• Experience with Kubernetes / EKS pod operations
• Strong Linux/ Unix fundamentals log management, performance debugging

Preferred Experience

• Experience with log pipeline tooling FluentD, Open Telemetry
• Familiarity with Splunk, New Relic, OpenSearch, Elasticsearch for log storage and search
• Experience building internal developer platforms or tooling CLI tools, internal APIs, automation frameworks
• Familiarity with MCP (Model Context Protocol) server development or AI Gateway architecture

Benefits

Pay is based on several factors including but not limited to education, work experience, certifications, etc. In addition to your salary, Blackhawk Network offers benefits including 401k with employer match, medical, dental, vision, 12 paid holidays throughout the year 2025, sick pay accrual according to state law, parental leave, life insurance, disability insurance, accident and illness insurance, health and dependent care flexible spending accounts, wellness benefits, and flexible time off for all full-time employees.

EEO Statement

Blackhawk Network provides equal employment opportunities to all employees and applicants for employment and prohibits discrimination and harassment of any type without regard to race, color, religion, age, sex, national origin, disability status, genetics, protected veteran status, sexual orientation, gender identity or expression, or any other characteristic protected by federal, state or local laws. Blackhawk Network believes that diversity leads to strength. This policy applies to all terms and conditions of employment, including recruiting, hiring, placement, promotion, termination, layoff, recall, transfer, leaves of absence, compensation, and training.