Principal Engineer – Agentic AI for IT Operations

About the role

The Opportunity At Ally, you get a startup feel, but experience the benefits of a company that has worked out the kinks and is fulfilling its purpose. We are always evolving and see that as a good thing. From owning our work to seeing its impact in the real world, our team is relentless in finding new ways technology can help make experiences better and help people. We are problem solvers, we value diverse thinking, we support one another, and we challenge ourselves to think bigger in the journey to deliver customer-obsessed tech solutions.

Responsibilities

• Develop, implement, and evolve agentic AI solutions that automate incident management, change management, and problem management workflows.
• Design and implement scalable integration patterns with monitoring, logging, ticketing, automation, and observability platforms.
• Partner with production support, SRE, and application teams to improve reliability, reduce MTTR, and automate operational runbooks.
• Establish engineering standards for safe deployment, testing, and monitoring of AI-enabled automation.
• Provide technical leadership during major incidents, ensuring agent-driven tooling and guidance supports consistent recovery.
• Drive completion of corporate tech risk and compliance mandates across application teams.
• Define technical controls, ensure alignment with regulatory requirements, and guide teams through implementation.
• Track program execution, identify gaps, and escalate risks with clarity and data.
• Serve as a key technical liaison across Security, Enterprise Operations, Infrastructure, Risk, and Compliance.
• Lead technical discussions, influence architectural decisions, and gain alignment across multiple stakeholder groups.
• Communicate complex concepts in a clear, approachable, and outcomes-focused way.
• Mentor engineers, champion best practices, and set a high bar for engineering quality and accountability.
• Foster a collaborative, approachable, team-first culture—modeling communication, ownership, and transparency.
• Shape long-term strategies for AI-driven IT operations across the enterprise.

Requirements

• Minimum Qualification: 7+ years of relevant experience or equivalent combination of education and experience.
• Preferred Qualifications: 5+ years software engineering experience with strong depth in distributed systems, enterprise operations, or SRE. Bachelors Degree in Computer Science or similar field or equivalent experience.
• Hands-on experience building automation or AI-driven operational tooling (e.g., agent-based systems, LLM apps, observability-driven automation, self-healing systems).
• Strong understanding of governance, risk, and compliance frameworks (technology control standards, audit readiness, policy implementation).
• Proven ability to lead cross-functional technical initiatives with diverse partners.
• Excellent communication skills; approachable, practical, and team-oriented working style.
• Ability to influence without authority and drive execution at scale.
• Awareness of the latest on AI technology.
• Experience with LLM frameworks, multi-agent orchestration, or retrieval-augmented automation.
• Experience with Claude Code, IDEs with AI agents like Cursor, VSCode with Agents etc..
• Familiarity with ServiceNow, APM/log platforms (Dynatrace, Splunk) CI/CD, and cloud platforms.
• Background in FinTech or regulated industries.

Skills

• Deep technical expertise in distributed systems, enterprise operations, or SRE.
• Expertise with incident, change, and problem management processes in large enterprise environments.
• Strong understanding of governance, risk, and compliance frameworks.
• Proven ability to lead cross-functional technical initiatives.
• Excellent communication skills.
• Ability to influence without authority and drive execution at scale.
• Knowledge of LLM frameworks, multi-agent orchestration, or retrieval-augmented automation.
• Experience with Claude Code, IDEs with AI agents like Cursor, VSCode with Agents etc..
• Familiarity with ServiceNow, APM/log platforms (Dynatrace, Splunk) CI/CD, and cloud platforms.
• Background in FinTech or regulated industries.

Benefits

All work requirements are subject to change at any time based on leader discretion and/or business need. The Work Itself Agentic AI Engineering & Architecture Architect, develop, and evolve agentic AI solutions that automate incident management, change management, and problem management workflows. Build intelligent agents for detection, triage, diagnostics, communication, and remediation across production applications for SaaS, COTS and homegrown applications Design and implement scalable integration patterns with monitoring, logging, ticketing, automation, and observability platforms. Operational Excellence & Production Support Partner with production support, SRE, and application teams to improve reliability, reduce MTTR, and automate operational runbooks. Establish engineering standards for safe deployment, testing, and monitoring of AI-enabled automation. Provide technical leadership during major incidents, ensuring agent-driven tooling and guidance supports consistent recovery. Governance, Risk, and Compliance Leadership Drive completion of corporate tech risk and compliance mandates across application teams. Define technical controls, ensure alignment with regulatory requirements, and guide teams through implementation. Track program execution, identify gaps, and escalate risks with clarity and data. Cross-Functional Partnership Serve as a key technical liaison across Security, Enterprise Operations, Infrastructure, Risk, and Compliance. Lead technical discussions, influence architectural decisions, and gain alignment across multiple stakeholder groups. Communicate complex concepts in a clear, approachable, and outcomes-focused way. Leadership & Team Culture Mentor engineers, champion best practices, and set a high bar for engineering quality and accountability. Foster a collaborative, approachable, team-first culture—modeling communication, ownership, and transparency. Help shape long-term strategies for AI-driven IT operations across the enterprise.

Pay

Base Pay Range: $90000 - $150000 USD An individual's position in the range is determined by the specific role, the scope and responsibilities of the role, work experience, education, certification(s), training, and additional qualifications. We review internal pay, the competitive market, and business environment prior to extending an offer.

Schedule

Work Schedule: Ally designates roles as (1) fully on-site, (2) hybrid, or (3) fully remote. Hybrid roles are generally expected to be in the office a certain number of days per week as indicated by your manager. Your hiring manager will discuss this role's specific work requirements with you during the hiring process. All work requirements are subject to change at any time based on leader discretion and/or business need.