Senior Software Engineer, AI

Essential Functions

• Subject Matter Expert for the design, development, maintenance, and deployment of AI agent systems, including one or more of LangGraph StateGraph patterns, CrewAI multi-agent orchestration, LlamaIndex data/agent workflows, or related frameworks.
• Design, build, and maintain FastAPI-based agent servers, including async/await endpoints, streaming responses, and health check endpoints.
• Implement comprehensive testing strategies — unit, integration, edge-case, and performance — for agents and services to ensure production reliability.
• Build and enforce guardrails for AI interactions, including input/output validation, safety checks, prompt hardening, policy enforcement, and robust errors handling in alignment with organizational security and compliance standards.
• Implement agent observability to include per-step traces, tool-call telemetry, cost/latency budgets, and SLO-based alerting.
• Design and build agent evaluation sets, adversarial tests, regression tests, and monitoring for safety/quality drift.
• Design and build RAG/retrieval pipelines using vector databases such as Chroma, Milvus, Weaviate, and Qdrant.
• Consult and collaborate with SMEs across Platform Engineering, Data Science, Security, Quality, and Product to ensure agent systems meet technical and operational requirements.
• Create, modify, and maintain AI system documentation, including architecture decision records (ADRs), runbooks, API specifications, network topology diagrams, data flow diagrams, and best-practice guides, maintained in a compliant and auditable state.
• Influence functional groups regarding operability, technical feasibility, engineering design, security posture, maintainability, and project schedule and documentation requirements.
• Aid with the planning and implementation of AI agent-based projects and solutions in collaboration with cross-functional teams, including determination and delivery on cost, timeline, and quality to ensure sustained production operations.
• Lead containerized deployment, start-up, commissioning, and release qualification activities using Docker multi-stage builds, dependency management, and container health check best practices.
• Provide technical mentoring and oversight to less experienced engineers responding to and troubleshooting AI agent and service-related issues.
• Create, edit, and adhere to Standard Operating Procedures (SOPs), process improvements, and standardization of engineering templates for agent development.
• Monitor agent performance and reliability, collect metrics and logs, and use data to identify and prioritize optimization opportunities.
• Perform ad-hoc and cross-functional projects assigned to support business needs and provide developmental opportunities.

Knowledge, Skills, & Abilities

• Excellent interpersonal skills and ability to work effectively in a cross-functional team environment.
• Excellent technical writing, communication, and organizational skills, including the ability to produce concise READMEs, architecture documents, and runbooks.
• Strong knowledge of engineering principles required to support secure, production-grade AI services and infrastructure.
• Deep technical knowledge and hands-on experience with AI agent frameworks, orchestration patterns, and context-management strategies.
• Strong leadership skills, with the ability to mentor junior engineers and drive technical standards.
• Strong knowledge and proficiency in data analytics, monitoring, and observability tooling (metrics, logging, tracing, dashboards) for production AI services.
• Solid understanding of API design principles, error handling patterns, and security best practices for backend services.
• Familiarity with AI safety and guardrail patterns, including prompt hardening and content filtering methodologies.

Working Environment / Physical Environment

• This position will either work onsite in Dunkirk, NY.
• Regular work schedule is Monday – Friday, within standard business hours.
• Flexibility is available with manager approval.
• Must possess mobility to work in a standard office setting and to use standard office equipment, including a computer.
• Lift and carry materials weighing up to 20 pounds.
• This position requires extensive use of computer keyboards and monitors.