Principal Architect

What You Will Do

• Lead the end-to-end design, development, and delivery of machine learning and Generative AI (GenAI) solutions, leveraging Databricks, Apache Spark, SQL, and Python for scalable data processing, feature engineering, and model development from problem framing to production deployment and business impact realization.
• Act as an Architect for large-scale Data Engineering and ML/GenAI initiatives, driving architecture decisions across lakehouse platforms (Databricks), distributed compute (Spark), and cloud ecosystems (AWS/GCP/Azure) to ensure scalability, reliability, and long-term maintainability.
• Design and implement advanced data pipelines and AI systems, including batch and streaming data processing (Spark), data modeling (SQL), and ML workflows (Python), along with multi-agent architectures, reasoning workflows, tool integration, and autonomous decision-making systems.
• Build and optimize robust data foundations for AI by developing high-quality, scalable ETL/ELT pipelines in Databricks, ensuring data availability, consistency, and performance for downstream ML/GenAI use cases.
• Define and institutionalize evaluation, validation, and governance frameworks for ML/GenAI systems, including model performance tracking, prompt evaluation, safety guardrails, hallucination mitigation, and compliance.
• Partner directly with business stakeholders and product leaders to translate objectives into data-driven AI/ML solutions, ensuring measurable value through well-defined data pipelines, KPIs, and experimentation frameworks.
• Establish and enforce best practices in MLOps, LLMOps, DataOps, and DevOps, including CI/CD pipelines, Databricks workflows, monitoring, observability, reproducibility, and cost optimization.
• Architect and oversee scalable cloud-based data and AI platforms, integrating Databricks Lakehouse, Spark processing layers, and cloud-native services for unified analytics and AI workloads.
• Drive experimentation strategy, including A/B testing, prompt optimization, and data-driven iteration, leveraging SQL analytics and Python-based experimentation frameworks.
• Provide mentorship to L4 and L5 engineers in data engineering (Spark, SQL, Databricks) and AI/ML development (Python, GenAI frameworks), including design reviews, code reviews, and career guidance.
• Lead cross-functional collaboration across data engineering, data science, platform engineering, and business teams to deliver integrated, production-grade AI solutions.
• Stay at the forefront of advancements in data engineering (Spark ecosystem, lakehouse architectures) and Generative AI/agentic systems, driving adoption of new technologies and best practices.

Basic Qualifications

• Doctorate degree and 2 years of experience OR
• Master's degree and 4 years of experience OR
• Bachelor's degree and 6 years of experience OR
• Associate's degree and 10 years of experience OR
• High school diploma / GED and 12 years of experience

Preferred Qualifications

• Deep expertise in machine learning, deep learning, and Generative AI (LLMs, transformers, embeddings, fine-tuning techniques).
• Proven track record of leading and delivering production-grade ML/GenAI systems end-to-end with measurable business impact with strong experience in designing scalable system architectures for ML and GenAI, including distributed systems and high-throughput pipelines.
• Expertise in MLOps/LLMOps ecosystems (MLflow, Kubeflow, Airflow, CI/CD, Docker, Kubernetes).
• Strong system design, architecture, and problem-solving skills with the ability to operate independently and lead large initiatives.
• Demonstrated proficiency in leveraging cloud platforms (AWS, Azure, GCP) for data engineering solutions. Strong understanding of cloud architecture principles and cost optimization strategies.
• Proven ability to mentor and guide junior and mid-level engineers (L4/L5).

Good-to-Have Skills

• Cloud certifications (AWS, Azure, or GCP) are a plus
• Strong experience with big data ecosystems, including Apache Spark, Hadoop, and large-scale distributed data processing
• Deep expertise in data engineering, including building and optimizing scalable data pipelines and platforms using Databricks, Spark, SQL, and Python
• Advanced proficiency in Python and modern ML/AI frameworks (e.g., PyTorch, TensorFlow, Hugging Face, LangChain, or similar)
• Experience designing robust evaluation and validation frameworks, including automated evaluations, human-in-the-loop systems, safety testing, and monitoring
• Extensive experience with Retrieval-Augmented Generation (RAG) architectures, vector databases, and knowledge-grounded AI systems
• Strong understanding of agentic AI frameworks, including orchestration, planning, memory management, and tool integration
• Solid foundation in statistical modeling, experimentation design (A/B testing), and causal inference
• Experience with NLP, semantic search, embeddings, and vector search systems
• Familiarity with Responsible AI practices, including fairness, explainability, governance, and regulatory compliance
• Hands-on experience with cloud-native AI/ML services across AWS, Azure, or GCP, including cost and performance optimization
• Experience with the Databricks Lakehouse platform for enterprise-scale data engineering, ML, and GenAI workloads
• Exposure to advanced evaluation techniques such as red-teaming, adversarial testing, and synthetic data generation
• Strong experience in data modeling and performance tuning for both OLAP and OLTP systems
• Hands-on experience with workflow orchestration tools such as Apache Airflow, and distributed processing frameworks like Apache Spark

Benefits

• Comprehensive employee benefits package, including a Retirement and Savings Plan with generous company contributions, group medical, dental and vision coverage, life and disability insurance, and flexible spending accounts
• A discretionary annual bonus program, or for field sales representatives, a sales-based incentive plan
• Stock-based long-term incentives
• Award-winning time-off plans
• Flexible work models where possible