Senior Software Engineer AI

Work, Play and Grow at BlackLine!

Responsibilities

Lead data pipeline development: Build and maintain PySpark ETL pipelines with high data quality and performance.

Manage integrations: Establish robust connections to client data sources via APIs and tools like FiveTran, Plaid, and BlackLine’s own internal connector ecosystem.

Ensure reliability: Monitor pipeline performance, automate testing, and validate data accuracy.

Optimize for scale: Implement performance improvements (e.g., CDC mechanisms, indexing strategies) for large-scale datasets.

Collaborate & innovate: Work with business stakeholders to refine data requirements and integrate cutting-edge AI and big data technologies.

You'll Get To

• Partner with data science, security, and product teams to set evaluation and governance standards (Guardrails, Bias, Drift, Latency SLAs).
• Mentor senior engineers and drive design reviews for ML pipelines, model registries, and agentic runtime environments.
• Lead incident response and reliability strategies for ML/AI systems.
• Collaborate with development teams to integrate AI solutions into existing workflows and applications.
• Define and manage MCP Registry for agentic component onboarding, lifecycle versioning, and dependency governance.
• Build CI/CD pipelines automating LLM agent deployment, policy validation, and prompt evaluation of workflows.
• Create scalable observability systems—tracking conversation outcomes, factual accuracy, latency, escalation patterns, and safety events.
• Implement logging, metering, and auditing for agent behavior, function calls, and compliance alignment.
• Develop and operationalize experimentation frameworks for agent evaluations, scenario regression, and performance analytics.
• Implement security measures to protect machine learning systems and data.
• Enforce secure deployment patterns with Infrastructure-as-Code and cloud-native secrets management.
• Define SLAs, error budgets, and compliance reporting mechanisms for ML and AI systems.

What You'll Bring

• 3+ years of experience with programming skills in languages such as Python, Java, or Scala.
• Expertise in ML frameworks (TensorFlow, PyTorch, scikit-learn) and orchestration tools (Airflow, Kubeflow, Vertex AI, MLflow).
• Proven experience operating production pipelines for ML and LLM-based systems across cloud ecosystems (GCP, AWS, Azure).
• Deep familiarity with LangChain, LangGraph, ADK or similar agentic system runtime management.
• Strong competencies in CI/CD, IaC, and DevSecOps pipelines integrating testing, compliance, and deployment automation.
• Hands-on with observability stacks (Prometheus, Grafana, Newrelic) for model and agent performance tracking.
• Understanding of governance frameworks for Responsible AI, auditability, and cost metering across training and inference workloads.
• Proficiency in containerization technologies (e.g., Docker, Kubernetes).

What We're Even More Excited If You Have

• Operations and Infrastructure: Proficient in scripting languages (e.g., Bash, python) for automation.
• Experience with workflow orchestration tools (e.g., Apache Airflow).
• Expertise in managing and optimizing cloud-based infrastructure.
• Familiarity with DevOps practices and tools for automated deployment.
• Understanding of network configurations and security protocols.
• Problem-solving and Critical Thinking: Ability to define problems, collect and analyze data, and propose innovative solutions.
• Strong critical thinking skills to evaluate models, identify limitations, and Adaptability and Learning Agility: Comfortable working in a fast-paced, rapidly evolving environment.
• Proactive in staying up to date with the latest trends, techniques, and technologies in AI/data science.

Pay Transparency Statement

Placement within this range depends upon several factors, including the applicant's prior relevant job experience, skill set, and geographic location. In addition to base pay, BlackLine also offers short-term and long-term incentive programs, based on eligibility, along with a robust offering of benefit and wellness plans.