Lead Engineer, AI Quality
Circle · Wilmington, DE · 3 wk ago
RemoteRemoteQuality Assurance$170k/yrFull-time
We need to output HTML fragment with sections: About Us? The instruction: Use for section headings (e.g. About the role, Responsibilities, Requirements, Qualifications, Skills, Benefits, Pay, Schedule). Do NOT repeat the job title as a heading. The posting includes "About Us" and "About The Role". We need sections only when posting has real specific content. We have "About Us" content describing company. Should we include a section heading "About Us"? The list of example headings didn't include About Us, but we can still use
for section headings (e.g. About the role, Responsibilities, Requirements, Qualifications, Skills, Benefits, Pay, Schedule). Do NOT repeat the job title as a heading. The posting includes "About Us" and "About The Role". We need sections only when posting has real specific content. We have "About Us" content describing company. Should we include a section heading "About Us"? The list of example headings didn't include About Us, but we can still use for any section heading. The instruction: "use for section headings (e.g. About the role, Responsibilities, Requirements, Qualifications, Skills, Benefits, Pay, Schedule)." It suggests those are examples, not exclusive. So we can include About Us heading.
We must not include boilerplate like EEO, etc. Also remove duplicated lines, navigation, cookie, legal, etc. Also remove "Learn more Candidate Safety & Interview Process NoticeDiversity, Equity & InclusionHow We Use Candidate DataEqual Employment OpportunityVisit our Candidate Hub..." etc.
We need to preserve all substantive information.
Let's extract content:
About Us Circle is building the world’s leading AI-powered, all-in-one platform for digital businesses. We make it possible for creators, coaches, educators, and businesses to bring together their audience with engaging discussions, live streams, events, chat, courses, and payments — all in one place, all under their own brand. We’re proud to be a fully remote company of around 270 (and growing!) team members from 30+ countries around the world. We seek exceptional individuals around the world, set them up to do the best work of their lives, and in turn, create a meaningful impact in their own lives. We don't track hours, but we do manage for high expectations very closely. We collaborate across time zones, are highly async, and like to document a lot. Twice a year, we bring the whole company together in beautiful places around the world for our company offsites. So far, we’ve hosted offsites in Turkey, Portugal, Mexico, Thailand, Colombia, Italy, Ireland, and more, with still more to come! Check out our Careers page for more about working at Circle.
About The Role The AI Quality engineering team at Circle owns the foundation for measuring, diagnosing, and improving the quality of Circle's AI-powered features. This team focuses on building the infrastructure to measure, diagnose, and improve production AI systems, rather than ML research or model training. We're looking for a Lead Engineer to help us build out the evaluation frameworks, observability tooling, and diagnostic infrastructure that tell us whether our AI Agents are working well, where to improve them, and how to make them faster and more cost-efficient. This is a hands-on player-coach role. You’ll also lead and manage the AI Quality engineering team, with an ambitious and growing roadmap. If you're excited about making AI systems work reliably, efficiently, and at scale, this is for you.
What You'll Be Doing Build and own our evaluation infrastructure. Design the CI/CD pipelines, scorers, and datasets that tell us whether Circle's AI agents (planners, tool-callers, and sub-agents) are actually working, from a single tool call to a full multi-turn conversation. Diagnose exactly where quality breaks down. Trace failures across the agent pipeline including plan creation vs. execution, tool selection, tool trajectory in complex areas like workflows, site builder, and analytics. Turn what you find into prototypes that solve the issue or identify priorities for AI core engineering can help. Grow the datasets that make evaluation possible. Stand up annotation workflows and build out AI-generated and simulated conversations so we can cover more of the product faster than manual labeling alone. Run structured experiments across prompts, models, and the agent harness. Evaluate new and open-source models against our production baseline, build the framework we use to decide when to shift models, and chase cost and latency wins through model swaps, caching, and routing by plan complexity. Lead and grow the AI Quality engineering team. Set technical direction and manage day-to-day priorities, all while staying hands-on in the code yourself. Partner closely with AI Core engineering. Work with the engineers building Circle's AI products so they have real confidence that their changes are actually improving quality, not just shipping.
What You'll Need To Be Successful 7+ years of experience building and shipping production software, ideally including LLM-powered agents that take real actions in a product. You've worked on complex, tool-using systems (multiple tools, planning or orchestration, sub-agents) not just simple, single-turn assistants, and you can walk us through something you shipped and how you knew it was actually working. Comfortable in Ruby on Rails / Python or ready to pick them up quickly. Ruby on Rails is our production system and the foundation that Circle’s AI Agents are built on and proficiency in Python is a strong plus, especially for the data and evaluation side of the work. Experience building evaluation or observability infrastructure for ML/AI systems. You've built eval pipelines, scorers, dashboards, or CI/CD for evals before and have experience with evaluation frameworks like Braintrust, LangSmith, or similar. Experience designing datasets, annotation workflows, or labeling pipelines for ML/AI evaluation. You know how to turn raw examples into a dataset you can trust through human annotation, AI-generated data, or simulation. Learn fast, experiment aggressively, and thrive in a highly dynamic environment. You’ll need to be comfortable running a lot of experiments and letting the results settle the argument, including the ones that don't work (which will be many of them in the beginning). Comfortable in a fast-paced environment with ambiguity. You’ll need to learn and pick up new technologies when projects require it. Strong alignment with our company values. You are proficient in English (spoken, written, and reading) at a CEFR Level C2 / ILR Level 5.
Compensation & Benefits Circle offers U.S.-benchmarked compensation globally, equity in the company with ongoing refresh grants, and 35 days of paid time off each year. We’re a remote-only team that comes together twice a year for company retreats in incredible destinations around the world. Alongside incredible flexibility and autonomy, we offer a benefits package that supports health, wellbeing, and professional growth. Learn more in our Candidate Hub.
Compensation Range: $170K
We need to include sections: About Us, About The Role (maybe "About the role"), Responsibilities (What You'll Be Doing), Requirements/Qualifications (What You'll Need To Be Successful), Benefits (Compensation & Benefits). Pay maybe separate but we have compensation range. Could combine Pay and Benefits. The instruction: include a section ONLY when posting has real specific content. We have compensation range $170K (maybe just that). Also benefits described. So we could have a Benefits section with details, and maybe a Pay subsection? But we can just have Benefits section covering both.
We must not create Pay/Schedule/Benefits sections unless posting actually states them. We have compensation and benefits, so we can include Benefits.
We should not include schedule (not mentioned). So omit.
We need to use - for any list of duties/requirements/benefits. So we need to turn the responsibilities into a list. Also requirements into a list. Benefits maybe list.
We need to preserve all substantive information; not invent or summarize away detail. So we need to keep details like specific areas (workflows, site builder, analytics). Also mention specific frameworks (Braintrust, LangSmith). Also mention specific numbers: 7+ years, 35 days PTO, $170K compensation, 270 team members, 30+ countries, offsites locations list.
We need to ensure we don't repeat job title as heading. The job title is Lead Engineer (AI Quality). Not to repeat.
We need to output only HTML fragment, no /, no markdown fences.
Let's structure:
About Us
... content ...
About the role
... content ...
Responsibilities
- Build and own our evaluation infrastructure. Design the CI/CD pipelines, scorers, and datasets that tell us whether Circle's AI agents (planners, tool-callers, and sub-agents) are actually working, from a single tool call to a full multi-turn conversation.
- Diagnose exactly where quality breaks down. Trace failures across the agent pipeline including plan creation vs. execution, tool selection, tool trajectory in complex areas like workflows, site builder, and analytics. Turn what you find into prototypes that solve the issue or identify priorities for AI core engineering can help.
- Grow the datasets that make evaluation possible. Stand up annotation workflows and build out AI-generated and simulated conversations so we can cover more of the product faster than manual labeling alone.
- Run structured experiments across prompts, models, and the agent harness. Evaluate new and open-source models against our production baseline, build the framework we use to decide when to shift models, and chase cost and latency wins through model swaps, caching, and routing by plan complexity.
- Lead and grow the AI Quality engineering team. Set technical direction and manage day-to-day priorities, all while staying hands-on in the code yourself. Partner closely with AI Core engineering. Work with the engineers building Circle's AI products so they have real confidence that their changes are actually improving quality, not just shipping.
Requirements
- 7+ years of experience building and shipping production software, ideally including LLM-powered agents that take real actions in a product.
- Experience with complex, tool-using systems (multiple tools, planning or orchestration, sub-agents) not just simple, single-turn assistants, and ability to walk through something shipped and how you knew it was actually working.
- Comfortable in Ruby on Rails / Python or ready to pick them up quickly. Ruby on Rails is our production system and the foundation that Circle’s AI Agents are built on; proficiency in Python is a strong plus, especially for the data and evaluation side of the work.
- Experience building evaluation or observability infrastructure for ML/AI systems. You've built eval pipelines, scorers, dashboards, or CI/CD for evals before and have experience with evaluation frameworks like Braintrust, LangSmith, or similar.
- Experience designing datasets, annotation workflows, or labeling pipelines for ML/AI evaluation. You know how to turn raw examples into a dataset you can trust through human annotation, AI-generated data, or simulation.
- Learn fast, experiment aggressively, and thrive in a highly dynamic environment. Comfortable running a lot of experiments and letting the results settle the argument, including the ones that don't work.
- Comfortable in a fast-paced environment with ambiguity. Willing to learn and pick up new technologies when projects require it.
- Strong alignment with our company values.
- Proficiency in English (spoken, written, and reading) at a CEFR Level C2 / ILR Level 5.
Benefits
- U.S.-benchmarked compensation globally.
- Equity in the company with ongoing refresh grants.
- 35 days of paid time off each year.
- Fully remote team with twice‑yearly company retreats in incredible destinations around the world.
- Flexibility and autonomy.
- Benefits package supporting health, wellbeing, and professional growth.
We also have compensation range: $170K. Should we include that as a separate Pay section? The instruction says include Pay section only if posting actually states them. It does state compensation range: $170K. So we could add a Pay section.
But we already have compensation in Benefits. Could separate Pay: Pay
Compensation range: $170K (U.S.-benchmarked globally).
However
for section headings (e.g. About the role, Responsibilities, Requirements, Qualifications, Skills, Benefits, Pay, Schedule)." It suggests those are examples, not exclusive. So we can include About Us heading.
We must not include boilerplate like EEO, etc. Also remove duplicated lines, navigation, cookie, legal, etc. Also remove "Learn more Candidate Safety & Interview Process NoticeDiversity, Equity & InclusionHow We Use Candidate DataEqual Employment OpportunityVisit our Candidate Hub..." etc.
We need to preserve all substantive information.
Let's extract content:
About Us Circle is building the world’s leading AI-powered, all-in-one platform for digital businesses. We make it possible for creators, coaches, educators, and businesses to bring together their audience with engaging discussions, live streams, events, chat, courses, and payments — all in one place, all under their own brand. We’re proud to be a fully remote company of around 270 (and growing!) team members from 30+ countries around the world. We seek exceptional individuals around the world, set them up to do the best work of their lives, and in turn, create a meaningful impact in their own lives. We don't track hours, but we do manage for high expectations very closely. We collaborate across time zones, are highly async, and like to document a lot. Twice a year, we bring the whole company together in beautiful places around the world for our company offsites. So far, we’ve hosted offsites in Turkey, Portugal, Mexico, Thailand, Colombia, Italy, Ireland, and more, with still more to come! Check out our Careers page for more about working at Circle.
About The Role The AI Quality engineering team at Circle owns the foundation for measuring, diagnosing, and improving the quality of Circle's AI-powered features. This team focuses on building the infrastructure to measure, diagnose, and improve production AI systems, rather than ML research or model training. We're looking for a Lead Engineer to help us build out the evaluation frameworks, observability tooling, and diagnostic infrastructure that tell us whether our AI Agents are working well, where to improve them, and how to make them faster and more cost-efficient. This is a hands-on player-coach role. You’ll also lead and manage the AI Quality engineering team, with an ambitious and growing roadmap. If you're excited about making AI systems work reliably, efficiently, and at scale, this is for you.
What You'll Be Doing Build and own our evaluation infrastructure. Design the CI/CD pipelines, scorers, and datasets that tell us whether Circle's AI agents (planners, tool-callers, and sub-agents) are actually working, from a single tool call to a full multi-turn conversation. Diagnose exactly where quality breaks down. Trace failures across the agent pipeline including plan creation vs. execution, tool selection, tool trajectory in complex areas like workflows, site builder, and analytics. Turn what you find into prototypes that solve the issue or identify priorities for AI core engineering can help. Grow the datasets that make evaluation possible. Stand up annotation workflows and build out AI-generated and simulated conversations so we can cover more of the product faster than manual labeling alone. Run structured experiments across prompts, models, and the agent harness. Evaluate new and open-source models against our production baseline, build the framework we use to decide when to shift models, and chase cost and latency wins through model swaps, caching, and routing by plan complexity. Lead and grow the AI Quality engineering team. Set technical direction and manage day-to-day priorities, all while staying hands-on in the code yourself. Partner closely with AI Core engineering. Work with the engineers building Circle's AI products so they have real confidence that their changes are actually improving quality, not just shipping.
What You'll Need To Be Successful 7+ years of experience building and shipping production software, ideally including LLM-powered agents that take real actions in a product. You've worked on complex, tool-using systems (multiple tools, planning or orchestration, sub-agents) not just simple, single-turn assistants, and you can walk us through something you shipped and how you knew it was actually working. Comfortable in Ruby on Rails / Python or ready to pick them up quickly. Ruby on Rails is our production system and the foundation that Circle’s AI Agents are built on and proficiency in Python is a strong plus, especially for the data and evaluation side of the work. Experience building evaluation or observability infrastructure for ML/AI systems. You've built eval pipelines, scorers, dashboards, or CI/CD for evals before and have experience with evaluation frameworks like Braintrust, LangSmith, or similar. Experience designing datasets, annotation workflows, or labeling pipelines for ML/AI evaluation. You know how to turn raw examples into a dataset you can trust through human annotation, AI-generated data, or simulation. Learn fast, experiment aggressively, and thrive in a highly dynamic environment. You’ll need to be comfortable running a lot of experiments and letting the results settle the argument, including the ones that don't work (which will be many of them in the beginning). Comfortable in a fast-paced environment with ambiguity. You’ll need to learn and pick up new technologies when projects require it. Strong alignment with our company values. You are proficient in English (spoken, written, and reading) at a CEFR Level C2 / ILR Level 5.
Compensation & Benefits Circle offers U.S.-benchmarked compensation globally, equity in the company with ongoing refresh grants, and 35 days of paid time off each year. We’re a remote-only team that comes together twice a year for company retreats in incredible destinations around the world. Alongside incredible flexibility and autonomy, we offer a benefits package that supports health, wellbeing, and professional growth. Learn more in our Candidate Hub.
Compensation Range: $170K
We need to include sections: About Us, About The Role (maybe "About the role"), Responsibilities (What You'll Be Doing), Requirements/Qualifications (What You'll Need To Be Successful), Benefits (Compensation & Benefits). Pay maybe separate but we have compensation range. Could combine Pay and Benefits. The instruction: include a section ONLY when posting has real specific content. We have compensation range $170K (maybe just that). Also benefits described. So we could have a Benefits section with details, and maybe a Pay subsection? But we can just have Benefits section covering both.
We must not create Pay/Schedule/Benefits sections unless posting actually states them. We have compensation and benefits, so we can include Benefits.
We should not include schedule (not mentioned). So omit.
We need to use - for any list of duties/requirements/benefits. So we need to turn the responsibilities into a list. Also requirements into a list. Benefits maybe list.
We need to preserve all substantive information; not invent or summarize away detail. So we need to keep details like specific areas (workflows, site builder, analytics). Also mention specific frameworks (Braintrust, LangSmith). Also mention specific numbers: 7+ years, 35 days PTO, $170K compensation, 270 team members, 30+ countries, offsites locations list.
We need to ensure we don't repeat job title as heading. The job title is Lead Engineer (AI Quality). Not to repeat.
We need to output only HTML fragment, no /, no markdown fences.
Let's structure:
About Us
... content ...
About the role
... content ...
Responsibilities
- Build and own our evaluation infrastructure. Design the CI/CD pipelines, scorers, and datasets that tell us whether Circle's AI agents (planners, tool-callers, and sub-agents) are actually working, from a single tool call to a full multi-turn conversation.
- Diagnose exactly where quality breaks down. Trace failures across the agent pipeline including plan creation vs. execution, tool selection, tool trajectory in complex areas like workflows, site builder, and analytics. Turn what you find into prototypes that solve the issue or identify priorities for AI core engineering can help.
- Grow the datasets that make evaluation possible. Stand up annotation workflows and build out AI-generated and simulated conversations so we can cover more of the product faster than manual labeling alone.
- Run structured experiments across prompts, models, and the agent harness. Evaluate new and open-source models against our production baseline, build the framework we use to decide when to shift models, and chase cost and latency wins through model swaps, caching, and routing by plan complexity.
- Lead and grow the AI Quality engineering team. Set technical direction and manage day-to-day priorities, all while staying hands-on in the code yourself. Partner closely with AI Core engineering. Work with the engineers building Circle's AI products so they have real confidence that their changes are actually improving quality, not just shipping.
Requirements
- 7+ years of experience building and shipping production software, ideally including LLM-powered agents that take real actions in a product.
- Experience with complex, tool-using systems (multiple tools, planning or orchestration, sub-agents) not just simple, single-turn assistants, and ability to walk through something shipped and how you knew it was actually working.
- Comfortable in Ruby on Rails / Python or ready to pick them up quickly. Ruby on Rails is our production system and the foundation that Circle’s AI Agents are built on; proficiency in Python is a strong plus, especially for the data and evaluation side of the work.
- Experience building evaluation or observability infrastructure for ML/AI systems. You've built eval pipelines, scorers, dashboards, or CI/CD for evals before and have experience with evaluation frameworks like Braintrust, LangSmith, or similar.
- Experience designing datasets, annotation workflows, or labeling pipelines for ML/AI evaluation. You know how to turn raw examples into a dataset you can trust through human annotation, AI-generated data, or simulation.
- Learn fast, experiment aggressively, and thrive in a highly dynamic environment. Comfortable running a lot of experiments and letting the results settle the argument, including the ones that don't work.
- Comfortable in a fast-paced environment with ambiguity. Willing to learn and pick up new technologies when projects require it.
- Strong alignment with our company values.
- Proficiency in English (spoken, written, and reading) at a CEFR Level C2 / ILR Level 5.
Benefits
- U.S.-benchmarked compensation globally.
- Equity in the company with ongoing refresh grants.
- 35 days of paid time off each year.
- Fully remote team with twice‑yearly company retreats in incredible destinations around the world.
- Flexibility and autonomy.
- Benefits package supporting health, wellbeing, and professional growth.
We also have compensation range: $170K. Should we include that as a separate Pay section? The instruction says include Pay section only if posting actually states them. It does state compensation range: $170K. So we could add a Pay section.
But we already have compensation in Benefits. Could separate Pay: Pay
Compensation range: $170K (U.S.-benchmarked globally).
However
About Us
... content ...
About the role
... content ...
Responsibilities
- Build and own our evaluation infrastructure. Design the CI/CD pipelines, scorers, and datasets that tell us whether Circle's AI agents (planners, tool-callers, and sub-agents) are actually working, from a single tool call to a full multi-turn conversation.
- Diagnose exactly where quality breaks down. Trace failures across the agent pipeline including plan creation vs. execution, tool selection, tool trajectory in complex areas like workflows, site builder, and analytics. Turn what you find into prototypes that solve the issue or identify priorities for AI core engineering can help.
- Grow the datasets that make evaluation possible. Stand up annotation workflows and build out AI-generated and simulated conversations so we can cover more of the product faster than manual labeling alone.
- Run structured experiments across prompts, models, and the agent harness. Evaluate new and open-source models against our production baseline, build the framework we use to decide when to shift models, and chase cost and latency wins through model swaps, caching, and routing by plan complexity.
- Lead and grow the AI Quality engineering team. Set technical direction and manage day-to-day priorities, all while staying hands-on in the code yourself. Partner closely with AI Core engineering. Work with the engineers building Circle's AI products so they have real confidence that their changes are actually improving quality, not just shipping.
Requirements
- 7+ years of experience building and shipping production software, ideally including LLM-powered agents that take real actions in a product.
- Experience with complex, tool-using systems (multiple tools, planning or orchestration, sub-agents) not just simple, single-turn assistants, and ability to walk through something shipped and how you knew it was actually working.
- Comfortable in Ruby on Rails / Python or ready to pick them up quickly. Ruby on Rails is our production system and the foundation that Circle’s AI Agents are built on; proficiency in Python is a strong plus, especially for the data and evaluation side of the work.
- Experience building evaluation or observability infrastructure for ML/AI systems. You've built eval pipelines, scorers, dashboards, or CI/CD for evals before and have experience with evaluation frameworks like Braintrust, LangSmith, or similar.
- Experience designing datasets, annotation workflows, or labeling pipelines for ML/AI evaluation. You know how to turn raw examples into a dataset you can trust through human annotation, AI-generated data, or simulation.
- Learn fast, experiment aggressively, and thrive in a highly dynamic environment. Comfortable running a lot of experiments and letting the results settle the argument, including the ones that don't work.
- Comfortable in a fast-paced environment with ambiguity. Willing to learn and pick up new technologies when projects require it.
- Strong alignment with our company values.
- Proficiency in English (spoken, written, and reading) at a CEFR Level C2 / ILR Level 5.
Benefits
- U.S.-benchmarked compensation globally.
- Equity in the company with ongoing refresh grants.
- 35 days of paid time off each year.
- Fully remote team with twice‑yearly company retreats in incredible destinations around the world.
- Flexibility and autonomy.
- Benefits package supporting health, wellbeing, and professional growth.
Pay
Compensation range: $170K (U.S.-benchmarked globally).
However