Sr. Data Engineer

Essential Job Duties

• Collaborate with data analysts, data scientists, ML engineers, software engineers, and business stakeholders to enable effective use of core data assets
• Design, develop, and maintain scalable data ingestion and transformation pipelines using Python, SQL, and modern data tooling
• Build and optimize data lake, lakehouse, warehouse, and data mart architectures
• Develop and maintain data models including facts, dimensions, feature datasets, and domain-specific data products
• Translate business requirements into design documents (e.g., ERDs, data flow diagrams, data models, and ML feature pipelines)
• Design and manage cloud-based data and ML infrastructure (Databricks preferred), including development, staging, and production environments
• Design, build, and operationalize machine learning pipelines for training, validation, deployment, and observability (e.g., performance, drift, reliability)
• Support ML model lifecycle management, including versioning, reproducibility, and lineage
• Develop and maintain ML feature stores and reusable feature pipelines for ML models
• Build and integrate AI-powered applications and agentic workflows (e.g., LLM-based agents, retrieval-augmented generation systems, workflow automation agents)
• Design and implement data pipelines for AI systems, including unstructured data (text, logs, embeddings, vector stores)
• Develop and maintain unit, integration, and data quality tests
• Participate in peer code reviews, pull requests, and team coding standards
• Document data pipelines, ML pipelines, models, infrastructure, and standard operating procedures
• Define infrastructure as code and support CI/CD pipelines for data and ML systems
• Ensure data privacy, security, and access control best practices (including AI data governance considerations)
• Identify and implement improvements in efficiency, scalability, resilience, and performance
• Contribute to evolving data, ML, and AI platform architecture, tools, and best practices

What You Need to Get the Job Done

• Collaboration & Business Engagement
• Comfortable working cross-functionally with both technical and non-technical stakeholders
• Quickly learn new domains and technologies
• Core Technical Skills
• Strong Python development experience
• Advanced SQL development and query optimization skills
• Understanding of Databricks and large-scale data processing
• Experience building and scaling data pipelines using Databricks and PySpark
• Deep understanding of data lake, lakehouse, data warehouse, and data mart architectures
• Experience with data modeling across a variety of business domains
• Experience with modern data tooling (e.g., dbt or similar transformation frameworks)
• Knowledge of data formats, data patterns, and modeling best practices
• Experience with cloud platforms (AWS preferred)
• Experience with CI/CD pipelines in a data engineering environment
• Git-based development workflows
• AI, ML & MLOps Skills
• Hands-on experience with AI prompt and agent frameworks (e.g., Claude Code, Cursor, Windsurfer, or similar)
• Experience building AI agents and agentic workflows
• Familiarity with handling structured and unstructured data (e.g., text, logs, embeddings)
• Experience building or supporting machine learning pipelines in production
• Familiarity with AI and MLOps in Databricks
• Experience with ML feature engineering and feature stores
• Understanding of ML model lifecycle management, monitoring, and evaluation

What Will Make Us REALLY Love You

• Familiarity with common business metrics across multiple domains
• Exposure to business systems like Netsuite, Salesforce, Marketo, Zuora, Gainsight, or Pendo
• Experience building KPI frameworks or domain-specific models (e.g., attribution, funnel, retention, financial metrics)
• Experience with streaming data and change data capture (CDC)
• Experience with real-time ML inference systems