LLMOps Engineer

About the Role

Artmac Soft is seeking an LLMOps Engineer to join our team in Prosper, Texas. This role involves designing, deploying, and maintaining production-grade LLM and Generative AI applications, as well as implementing workflows for LLM deployment, model versioning, and prompt version management.

Responsibilities

• Design, deploy, and maintain production-grade LLM and Generative AI applications.
• Implement workflows for LLM deployment, model versioning, and prompt version management.
• Develop and maintain automated LLM evaluation pipelines for quality, accuracy, relevance, and safety.
• Implement LLM tracing and observability to monitor application behavior and model performance.
• Monitor production LLM applications for latency, reliability, errors, token usage, and overall performance.
• Work with vector databases to support semantic search, retrieval, and RAG-based applications.
• Optimize LLM applications for cost, latency, scalability, and throughput.
• Establish monitoring and alerting mechanisms for LLM and AI workloads.
• Troubleshoot production issues related to models, prompts, APIs, retrieval pipelines, and infrastructure.

Requirements

• 8+ years of experience in LLMOps, MLOps, DevOps, ML Engineering, or related areas.
• Hands-on experience with LLM deployment and productionization.
• Strong understanding of prompt engineering and prompt/model versioning.
• Experience with LLM evaluation, testing, and benchmarking.
• Familiarity with LLM observability, tracing, logging, and monitoring.
• Experience working with vector databases such as Pinecone, Weaviate, Milvus, Qdrant, Elasticsearch, or OpenSearch.
• Proficiency in Python and experience developing automation or backend workflows.
• Familiarity with REST APIs and LLM APIs such as OpenAI-compatible APIs, Azure OpenAI, Anthropic, or Google Gemini.
• Understanding of RAG pipelines and retrieval workflows.

Qualifications

Bachelor's degree or equivalent combination of education and experience.