AI SR Software Engineer

About the role

We are seeking an experienced professional to lead the configuration, implementation, and deployment an AI Gateway solution (e.g. litellm) on-premise in our AWS boundary. This role will focus on delivering a production-ready fully automated AI Gateway infrastructure with comprehensive documentation and initial rollout support.

Responsibilities

• Implementation & Configuration: Design and implement AI Gateway infrastructure using Infrastructure as Code (IaC); Configure and customize according to organizational requirements (security and standards); Deploy solution within our AWS environment; Conduct comprehensive testing including functional, performance, and integration testing; Ensure solution meets security, scalability, and reliability requirements.
• Documentation & Knowledge Transfer: Create detailed technical documentation for the implemented solution; Develop onboarding guides and operational runbooks; Document configuration standards and deployment procedures; Provide architecture diagrams and system documentation.
• Rollout Support: Support initial production rollout and deployment; Provide troubleshooting and issue resolution during launch phase; Conduct knowledge transfer sessions with internal teams; Assist with early adoption and usage optimization.

Required Skills & Experience

• AWS Expertise: Strong experience with AWS services and cloud architecture.
• Infrastructure as Code: Proficiency with IaC tools (Terraform, CloudFormation, CDK, or similar).
• AI Gateway Experience: Hands-on experience with LiteLLM or similar AI Gateway platforms.
• Demonstrated ability to implement and configure cloud-based infrastructure solutions.
• Experience with API management and gateway technologies.
• Strong documentation and technical writing skills.

Preferred Qualifications

• Security certifications or specialized security experience (AWS Security, cloud security best practices).
• Experience with AI/ML model deployment and management.
• Knowledge of rate limiting, token management, and API cost optimization.
• Familiarity with observability and monitoring tools (CloudWatch, Grafana).
• Experience with CI/CD pipelines and DevOps practices.

Deliverables

• Production-ready AI Gateway implementation.
• Complete IaC codebase with version control.
• Comprehensive technical documentation.
• Onboarding guides and operational procedures.
• Post-implementation support through initial rollout period.