AI Operations Engineer

About the role

We are seeking an AI Operations Engineer to help support and optimize our next-generation AI platform focused on compliance, surveillance, and intelligent workflow automation. This role will work closely with AI engineers, product managers, and domain experts to evaluate AI outputs, manage training and evaluation datasets, improve prompts and agent behaviors, and help operationalize LLM-powered systems across the platform.

Work Location

The ideal candidate is detail-oriented, analytical, and excited about working hands-on with modern AI technologies including agents, MCP servers, vector search, RAG pipelines, document/media processing, and large language models.

Responsibilities

• Test and evaluate AI agents, prompts, and workflows across multiple use cases
• Create, maintain, and curate labeled datasets for AI evaluation and tuning
• Review AI-generated outputs for quality, accuracy, safety, and consistency
• Update and optimize prompts, agent instructions, and AI “skills”
• Help improve retrieval quality for vector search and RAG pipelines
• Assist with evaluation frameworks and benchmark creation for LLM systems
• Work with engineering teams to validate MCP connectors and integrations
• Analyze model failures, hallucinations, and edge cases to improve system performance
• Participate in human-in-the-loop review workflows for AI-generated content
• Document testing procedures, findings, and recommended improvements
• Collaborate with compliance and domain experts to refine AI behaviors and detection logic

What working at ACA offers

• A competitive compensation package
• Total Rewards package including medical and dental coverage and 401(k) plans
• A wide range of paid time off options
• A flexible work environment
• Designated ACA Paid Holidays, Summer Fridays, Personal/Family Care, and other leaves of absence
• Unique benefits such as Student Debt Forgiveness and Pet Insurance

About ACA

ACA Group is the leading governance, risk, and compliance (GRC) advisor in financial services. We empower our clients to reimagine GRC and protect and grow their business. Our innovative approach integrates consulting, managed services, and our ComplianceAlpha® technology platform with the specialized expertise of former regulators and practitioners and our deep understanding of the global regulatory landscape.

What we commit to

ACA is firmly committed to a policy of nondiscrimination, which applies to recruiting, hiring, placement, promotions, training, discipline, terminations, layoffs, transfers, leaves of absence, compensation and all other terms and conditions of employment. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, age, disability or protected veteran status, or any other legally protected basis, in accordance with applicable law.