AI Automation Engineer
AppLab Systems, Inc · Bellevue, WA · 2 wk ago
On-siteEngineering$150k/yrContract
About the role
Position: AI Automation Engineer. Location: Bellevue, WA (Onsite). Openings: 1. Employment Type: Full-Time.
Responsibilities
- Design and build end-to-end AI testing frameworks covering unit, functional, integration, and regression testing, including automated test case generation and model output validation against defined quality benchmarks.
- Develop load and performance testing frameworks to simulate high-concurrency scenarios, validating scalability, latency SLAs, and platform stability under peak workloads.
- Build LLM evaluation pipelines measuring relevance, groundedness, and hallucination metrics, integrating them into CI/CD workflows to enforce automated quality gates before every model or agent deployment.
- Own the AI platform regression testing strategy by maintaining automated test suites that validate agent behavior, RAG pipeline accuracy, and API contracts after every release to prevent quality degradation.
Requirements
- Minimum 15 years of overall IT experience.
- 3+ years of hands-on AI/ML experience.
Skills
Mandatory Skills
- Strong experience with ETL Pipelines.
- Hands-on experience in Machine Learning / Artificial Intelligence.
- Experience with AI Operations, including MLOps, AgentOps, and LLMOps.
- Strong programming skills in Python and SQL.
- Experience in System Design.
- Hands-on experience with MCP (Model Context Protocol) Client-Server Design.
Preferred Skills
- AIOps / MLOps
- Observability for AI
- Bash Scripting
Pay
Salary: Up to $150K.