Lead Software Development Engineer in Test - AI Tooling & Automation

About the role

We are seeking a Software Quality Lead Member of Technical Staff (LMTS) to join our Revenue Action Orchestration (RAO) team. This individual will design, build, and maintain scalable, AI-first testing frameworks and automation suites to validate complex RAO workflows and agentic capabilities.

Responsibilities

• Translate high-level quality strategies into actionable automation plans, focusing heavily on validating LLM integrations, prompt accuracy, and deterministic system behaviors.
• Dig deep into the application code to understand system architecture, identify quality gaps, perform white-box testing, and actively contribute to fixing code and test flakiness.
• Implement and support evaluation processes for subjective quality dimensions of our AI features, including response relevance, user safety, and contextual accuracy.
• Partner closely with developers, product managers, and engineers within the RAO ecosystem to ensure quality is integrated early and continuously throughout the agile delivery lifecycle.

Requirements

• Strong Java Skills: Strong technical fundamentals of Java, particularly in data structures and algorithms.
• AI/LLM Validation Mindset: Hands-on experience evaluating Large Language Models (LLMs) for quality response, including familiarity with GPT-like models and AI evaluation frameworks is a big plus.
• Agile Delivery at Pace: Proven track record of delivering automation solutions in a fast-paced CI/CD environment, supporting rapid deployment schedules without compromising quality.
• Collaboration & Communication: Excellent communication skills, with the ability to work cross-functionally with product managers, engineers, AI/ML engineers, and data scientists.
• Problem-Solving & Analytical Skills: Strong analytical and problem-solving abilities with a focus on identifying quality gaps and driving improvement.

Qualifications

• A software engineer with a strong coding background and understanding of delivering software at pace in Agentic PDLC.
• Strong experience with white-box testing and fixing code is a huge plus.
• Experience working with public cloud infrastructure (AWS/Azure/GCP) and/or foundational knowledge of the Salesforce platform (Salesforce Certifications are a plus).
• Prior experience testing autonomous agents, conversational UIs, or generative AI workflows.

Skills

• Strong Java Skills
• AI/LLM Validation Mindset
• Agile Delivery at Pace
• Collaboration & Communication
• Problem-Solving & Analytical Skills

Benefits

At Salesforce, we offer a comprehensive benefits package including time off programs, medical, dental, vision, mental health support, paid parental leave, life and disability insurance, 401(k), and an employee stock purchasing program. For more details, visit this link.

Pay

The typical base salary range for this position is $172,500 - $260,100 annually. In select cities within the San Francisco and New York City metropolitan area, the base salary range for this role is $207,800 - $285,800 annually.

Schedule

This role is full-time.