Principal Software Engineer, Applied AI (Forward Deployed)

About the role

We are seeking a highly skilled and driven Principal Software Engineer with a strong background in full-stack development, particularly in backend technologies and Agentic AI, to join our AI/ML team.

This role combines hands-on model development with robust backend engineering and infrastructure work. You’ll help build scalable systems, support R&D initiatives, and ensure rapid iteration and deployment of machine learning solutions in dynamic, production-ready environments.

You will work embedded with our top clients and the client teams to conduct use case discovery with senior stakeholders directly, and develop solutions that address complex client needs. This will require being onsite with clients 3-4 days a week on a regular basis in New York or London.

What You’ll Do

• Contribute during a phase of rapid growth as we focus on scaling models and improving platform performance.
• Help build backend systems, support client-facing deployments, and enable smoother workflows for machine learning solutions.
• Develop and Maintain AI/ML Systems: Build robust, scalable backend systems that support machine learning operations and data processing pipelines.
• Cloud Operations and Management: Oversee and optimize cloud infrastructure to ensure efficient deployment and operation of ML models.
• Problem Solving: Independently explore and address complex problem spaces to improve system capabilities and performance without extensive guidance.
• Cross-Functional Collaboration: Work closely with ML engineers and data scientists to integrate advanced ML technologies, ensuring seamless operations across various platforms.
• Client Engagement: Collaborate directly with Invisible’s clients, working embedded with client teams to support use case discovery, product development, and AI deployment.
• Innovation and R&D: Actively participate in research and development of new tools that can enhance our AI capabilities and workflows.

What We Need

• 10+ years of software engineering experience, with a strong focus on ML engineering and deploying machine learning models in production.
• Extensive experience in full-stack development, particularly in backend environments that support AI/ML workloads.
• Prior experience working directly with clients in use case discovery, product development, and leading client engagements.
• Strong proficiency in Python, with deep expertise in LLMs, AI Agents, and ML model development.
• Experience designing and deploying scalable ML systems, such as retrieval-augmented generation (RAG) pipelines and production-grade AI applications.
• Extensive experience with cloud platforms (AWS, GCP, Azure) and operational best practices for ML workloads.
• Familiarity with Kubernetes and other container management tools.
• Ability to write well-structured, organized code and automated unit/E2E tests.
• Comfortable with polyglot persistence models (SQL vs. NoSQL).
• ML Operations: Experience with MLOps frameworks and best practices; familiarity with DevOps principles as applied to machine learning models, including model versioning, monitoring, and lifecycle management.
• Problem Solving: Ability to operate independently in unstructured environments, demonstrating a proactive and investigative approach to tackling challenges.
• Communication: Excellent communication skills, with the ability to collaborate effectively in dynamic, cross-functional teams, including data scientists, researchers, and software engineers.