Software Engineer II - Software Engineering # 26-18445

General Description

We are looking for a highly technical engineer to build and support weather data acquisition, processing, and analysis products, including our weather service, using a cloud-based solutions stack. We are open to early or advanced career candidates with strong examples of client contributions and highly independent work in a fast-paced software delivery environment.

Essential Attributes/Experience

• Excellent coding skills (Python) with production software deployment experience
• Big data experience (terabyte or petabyte level data sources)
• Core understanding of cloud computing (e.g., AWS services like IAM, Lambda, S3, RDS)

Responsibilities

• Architect and propose new AWS/Databricks solutions & updates to existing backend systems that ingest, cache, and process terabyte and petabyte level weather data (e.g., from IBM Weather Company API)
• Work closely with the product management team and end users to understand customer experience and system requirements, build backlog, and prioritize work
• Build infrastructure as code (e.g., Terraform)
• Improve system scalability and optimize workflows to reduce cloud costs
• Create and update REST APIs and backend processes (e.g., AWS Chalice/Lambda) for current, historical, and forecast weather data
• Build/support solutions involving containerization (e.g., Docker), caching (Redis), and databases (e.g., PostgreSQL/PostGIS)
• Support Databricks/Spark pipelines for historical weather data backfill into S3 (Parquet/Delta Lake) and data analysis (SQL/PySpark)
• Migrate and maintain CI/CD pipelines (e.g., GitHub Actions)
• Enhance monitoring and alerting for weather systems (e.g., Datadog)
• Enable field testing and customer support operations by debugging and fixing data issues
• Work with data scientists to scalably fetch and manipulate large weather data sets for models and analysis

Duration: 12 Months

Remote/Onsite: Position is likely to start onsite and could transition to remote depending on level of performance.