Data Platform Lead (Hybrid)

Responsibilities

• Serve as the platform owner for AMA’s enterprise Databricks lakehouse, accountable for its technical direction, delivery, and operational health.
• Translate the enterprise data platform strategy and multi-year roadmap into an actionable technical delivery plan and engineering backlog.
• Define and maintain the platform’s technical architecture, including workspace topology (development, staging, production), Delta Lake design, and Unity Catalog structure.
• Maintain a prioritized platform engineering backlog using agile delivery methods, balancing new capability development, technical debt remediation, and operational work.
• Evaluate emerging Databricks and data ecosystem capabilities and recommend adoption where they advance reliability, cost efficiency, or business value.
• Contribute to platform budget planning and forecasting, and provide technical input into platform-related procurement and vendor management.
• Own platform prioritization and backlog, including intake and sequencing of requests from business units and engineering teams.
• Own the technical onboarding of new data sources and business-unit workloads onto the platform.
• Collaborate with Sr Data Engineer to lead the design, build, and operation of data ingestion, transformation, and delivery pipelines across batch and streaming workloads.
• Ensure adherence engineering standards for data modeling, code quality, version control, testing, and CI/CD across the platform.
• Design and maintain self-service analytics capabilities and reusable data products that reduce duplicated effort across business units.
• Provide and operate shared MLOps platform capabilities (e.g., model registry, deployment frameworks), enabling data science and engineering teams to operationalize models.
• Develop and maintain platform documentation, runbooks, and standard operating procedures.
• Serve as the final decision authority on platform architecture and engineering standards; provide design approvals and guardrails for data engineering teams.
• Contribute to the reliability, performance, scalability, and security of the enterprise data platform, in partnership with the Director II and both IT and federated business-unit engineering teams.
• Define and operate observability tooling and monitoring practices for platform health, cost optimization, model performance, data quality, and governance compliance; drive continuous improvement initiatives.
• Drive cloud cost optimization through cluster configuration, capacity planning, storage management, and cost allocation across business units.
• Implement platform security controls, including encryption, access management, and audit logging, in alignment with IT Security policy and HIPAA and other regulatory requirements.
• Partner with the Data Governance Lead to translate Enterprise Data Office policies into platform configuration and automated enforcement.
• Implement technical controls supporting AI governance, including model lifecycle management, monitoring, and drift detection.

Requirements

• Bachelor’s degree in computer science, engineering, information systems, or related field preferred or equivalent work experience and HS diploma/equivalent education required
• 7+ years of experience in data engineering, data platform engineering, or data architecture required.
• Proven experience designing, building, and operating a large-scale data platform, preferably built on modern lakehouse architectures (Databricks preferred)
• Proven expertise with Delta Lake, Unity Catalog, Apache Spark, and SQL-based transformation frameworks (dbt or equivalent)
• Experience owning platform reliability, performance tuning, observability, and cloud cost optimization
• Proven experience implementing data governance controls, including access, data quality, metadata, and lineage, within a data platform
• Working knowledge of AI/ML workflows, MLOps practices, and model lifecycle management
• Experience operating within a regulated data environment (healthcare / HIPAA, financial services, or similar) preferred
• Demonstrated success leading technical teams and collaborating across engineering, security, and business disciplines
• Strong ability to influence, negotiate, and drive alignment across stakeholders without direct authority
• Strong ability to balance innovation and delivery speed with operational reliability, cost efficiency, and regulatory requirements