AI/ML Architect (Databricks, AWS)

Role Overview

We are seeking an experienced AI/ML Architect with deep hands-on expertise in Databricks on AWS to lead the design and implementation of scalable, high performance data and machine learning platforms. The ideal candidate combines architectural thinking with strong engineering execution, demonstrating the ability to build modern lakehouse systems, optimize large scale pipelines, and drive analytical and ML capabilities across the organization.

This role requires working with large, multi-terabyte datasets, advanced analytics, and end to end ML lifecycle management using Databricks, Python, PySpark, and AWS-native services.

Key Responsibilities

• Develop, train, and optimize ML models using Python, PySpark, MLflow, and Databricks Machine Learning.
• Conduct exploratory data analysis (EDA) to identify patterns, trends, and insights in large datasets.
• Deploy ML models into production using MLflow, Databricks Workflows, or other MLOps pipelines.
• Build analytics solutions such as forecasting, anomaly detection, segmentation, or recommendation systems.
• Design ML architectures aligned with Databricks Lakehouse on AWS.
• Architect and build scalable ETL/ELT pipelines using PySpark, SQL, and Databricks Workflows.
• Implement Delta Lake best practices, including OPTIMIZE, ZORDER, partitioning, and schema evolution.
• Design lakehouse layers (Bronze/Silver/Gold) with strong separation of compute and serving layers.
• Optimize cluster performance and jobs using Spark tuning, caching, and shuffle minimization.
• Ensure robust data availability for downstream ML and analytics workloads.
• AWS Cloud Integration Architect end-to-end data and ML solutions using AWS services, including: S3 for storage, IAM for identity & access, Glue Catalog for metadata management, Networking for secure, high throughput data movement.
• Integrate Databricks with AWS-native compute, API layers, and low-latency endpoints.
• Translate business problems into scalable analytical or ML architectures.
• Communicate complex statistical and architectural concepts to non-technical stakeholders.
• Collaborate with product, engineering, and business leaders to drive data-informed initiatives.
• Provide design leadership while remaining hands-on in execution.

Required Skills & Qualifications

• Bachelor’s or master’s in computer science, Data Science, Engineering, Statistics, or related field.
• 10+ years of experience in data engineering, ML engineering, or AI/ML architecture roles.
• Deep Expertise In Databricks On AWS, Including PySpark / Spark SQL, Databricks Notebooks, Delta Lake, Unity Catalog, MLflow, Databricks Jobs & Workflows.
• Strong programming ability in Python (pandas, numpy, scikit-learn).
• Demonstrated experience with large-scale, multi-terabyte data processing.
• Strong understanding of ML algorithms, distributed systems, and data optimization.

Preferred Experience

• With MLOps and production deployment pipelines.
• Strong grasp of AWS-native data and compute services.
• Familiarity with CI/CD using GitHub Actions, GitLab CI, or similar.
• Familiarity with deep learning frameworks (TensorFlow, PyTorch).

Key Competencies

• Strong analytical and problem-solving skills.
• Ability to work in fast-paced, highly collaborative environments.
• Excellent communication and presentation abilities.
• Self-driven with exceptional attention to architectural detail.