Senior ML Platform Engineer (Autonomous Driving)

About the role

We are looking for an AD ML Platform Engineer at 42dot. This position involves setting technical strategy, developing and managing large-scale data platforms, and collaborating with various teams to ensure alignment with the overall Autonomous Driving System Architecture.

Responsibilities

• Set technical strategy and oversee development of high scale, reliable data platform to manage, visualize and serve large-scale datasets for ML model training and validation.
• Build up the data lakehouse for autonomous driving scene datasets, including the sensor data, calibration data, as well as annotation data.
• Develop the Autonomous Driving Data SDK, focusing on scene data search, datasets preparation, dataset loading, etc.
• Dig into performance bottlenecks throughout the data processing pipelines, addressing issues related to data processing latency, data search latency, and Test Procedure (TP) coverage.
• Bootstrap and maintain infrastructure for Data Platform components, including Data Processing Pipeline, Database, Data Lakehouse, and Data Serving.
• Collaborate with cross-functional teams, such as ML algorithm, ML application, and Cloud Infra, to align ML Platforms with the overall Autonomous Driving System Architecture.

Qualifications

• Bachelor's degree or higher in Computer Science, Engineering, Robotics, or a similar technical field.
• Minimum of 7 years of experience in Data Engineering or ML Platform roles.
• Expert-level proficiency in Python and solid experience in Python SDK development.
• Solid working experience in Databases (e.g., MongoDB, PostgreSQL, etc).
• Strong understanding of modern AI frameworks (e.g., PyTorch, TensorFlow etc.), especially the principle of distributed data loader for model training.
• Hands-on experience with data pipeline job orchestration with Databricks Workflows or Apache Airflow, as well as integrating data pipelines with machine learning models.
• Extensive experience with data technologies and architectures such as Data Warehouse (e.g., Hive) or Lakehouse (e.g., Delta Lake).
• Experience with Apache Spark or other big data computing engines.
• Excellent leadership and communication skills, with a demonstrated ability to lead technical projects.

Preferred Qualifications

• Experience with autonomous vehicle sensor data (e.g., LiDAR, camera, radar).
• Experience with ML model training lifecycle (e.g., data preparation, model training / validation / deployment, etc).
• Understanding data governance principles, data privacy regulations, and experience implementing security measures to protect data.
• Understanding of Large Models, like VLM.