Data Engineer, Platform

About the role

The Data Engineer on the Platform team at Basis builds trustworthy data pipelines with comprehensive provenance and quality gates, curates documented datasets for training and evaluation, and ensures data infrastructure scales reliably. You will work on both platform-specific data needs and cross-project data coordination, preventing duplicate work and facilitating shared datasets.

Responsibilities

• Design and build data pipelines for training and evaluation across Basis research projects and platform offerings, ensuring reliability, performance, and scalability.
• Implement data quality frameworks including validation rules, quality gates, anomaly detection, and monitoring that catch data issues before they impact research or production systems.
• Develop and maintain feature stores or equivalent systems that enable consistent feature access across training and serving environments, preventing train-serve skew.
• Ensure data provenance and lineage tracking so researchers and engineers can understand data origins, transformations applied, and dependencies, enabling reproducible experiments and debugging.
• Curate documented datasets for model training and evaluation, including dataset versioning, comprehensive documentation, quality metrics, and metadata that enables appropriate usage.
• Curate documented datasets for model training and evaluation, including dataset versioning, comprehensive documentation, quality metrics, and metadata that enables appropriate usage.
• Coordinate cross-project data initiatives to prevent duplicate data work, facilitate shared datasets, and ensure consistent data practices across Basis as the organization scales.
• Optimize data infrastructure for scale as compute grows, including cost optimization, performance tuning, caching strategies, and efficient data access patterns.
• Collaborate with research and engineering teams to understand data needs, translate requirements into technical solutions, and provide consultation on data architecture and best practices.
• Implement data governance policies ensuring compliance with privacy regulations, security requirements, and responsible AI practices as Basis serves external customers.
• Contribute to the culture and direction of Basis by modeling data quality rigor, documentation excellence, and focus on trustworthy data infrastructure.

Requirements

• Have demonstrated significant achievements in data engineering for ML/AI systems.
• Possess strong proficiency in data technologies including SQL (expert level), Python for data processing, distributed computing frameworks (Spark, Dask), and workflow orchestration tools (Airflow, Dagster, Prefect).
• Experience with cloud data platforms including data warehouses (Snowflake, BigQuery, Redshift), data lakes, object storage (S3), and streaming systems (Kafka, Kinesis, Flink) for both batch and real-time processing.
• Understand ML data requirements including feature engineering, training/validation/test splits, data versioning, experiment reproducibility, and the specific data needs of different model types and training procedures.
• Be skilled at data quality and governance including implementing validation frameworks, anomaly detection, data lineage tracking, metadata management, and ensuring compliance with privacy and security policies.
• Have knowledge of data modeling principles for both relational and NoSQL systems, understanding of schema design, normalization/denormalization tradeoffs, and performance optimization.
• Value data provenance and documentation. You ensure data pipelines are transparent, decisions are documented, and others can understand and trust the data you deliver.
• Progress with autonomy on complex data challenges. You can scope data projects, make sound architectural decisions, and deliver complete solutions from ingestion through consumption.
• Be excited about enabling rigorous research through trustworthy data infrastructure that advances our ability to solve intractable problems.

Qualifications

• Experience with feature stores (Tecton, Feast) or building feature platforms.
• Background in ML research or research engineering providing understanding of data needs across experiment lifecycle.
• Experience with data lineage tools (Apache Atlas, DataHub, Monte Carlo) and metadata management.
• Knowledge of vector databases and embedding pipelines for modern AI applications.
• Contributions to data engineering open-source projects (Airflow, dbt, Great Expectations).
• Understanding of responsible AI and data governance practices.