Software Development Engineer in Test, ML

Responsibilities

• Design, develop, and maintain automated test frameworks, test scripts for ML inference services and related ML platform components using python/Java
• Build and automate realistic, representative test account creations and datasets to evaluate model behavior across different scenarios
• Validate score distributions, ranking behavior, output quality, and other model signals for incoming ML models prior to release
• Perform different types of testing, including functional, integration, regression and API testing working with RESTful APIs, microservices and databases, and non-functional performance and load testing to verify inference service scalability, latency requirements and reliability under production like conditions
• Work closely with ML engineers and ops teams to ensure that all features and bug fixes come with automated test coverage, ensuring continuous integration and deployment
• Debug and analyze test failures, model anomalies and report defects
• Drive high standards of quality for both engineering decisions and customer-facing features with optimal test and automation strategy
• Develop and implement best practices for test automation that is highly scalable and maintainable
• Monitor test execution and optimize performance in test environments
• Come up with innovative solutions to organizational problems

Required Qualifications

• Bachelor's degree or equivalent
• 3+ years of experience as an SDET, with strong experience testing backend systems
• Proficiency in Python/Java/Scala/go for test automation
• Hands-on experience with API testing tools (e.g., Postman or similar) for distributed systems
• Familiarity with CI/CD pipelines and test execution in Jenkins or similar environments such as jenkins, GitHub Actions, ArgoCD, etc
• Experience building and maintaining automated test frameworks (e.g., pytest, JUnit, TestNG, etc.)
• Experience with databases for backend validation
• Experience with cloud and container technologies (AWS, GCP, Kubernetes, Docker, etc.)
• Familiarity with monitoring and observability tools (e.g., Prometheus, Grafana, Datadog, etc.) for validating service health
• Strong understanding of software development lifecycle (SDLC) and agile methodologies
• Excellent problem-solving skills and excellent communication skills

Preferred Qualifications

• Experience testing machine learning systems, such as recommendation systems, search ranking systems, or other probabilistic ML application systems
• Experience creating synthetic or seeded test data to simulate realistic customer/account behaviors
• Experience evaluating model outputs using statistical techniques, distribution analysis, or scenario-based validation, rather than deterministic assertions
• Knowledge of data pipelines, feature stores, inference systems or model-serving infrastructure (Seldon, KServe, Ray Serve, etc.) is a plus
• Experience testing online services with high RPS and low latency requirements
• Experience with performance and load testing tools (Locust, k6, Gatling, JMeter, etc.)
• Experience with Databricks or similar ML platform tooling