Databricks Architect

About the role

PEMCO is seeking a Databricks Architect to join our community. The ideal candidate will lead the end-to-end architecture, governance, and scalability of the Azure Databricks Lakehouse platform at PEMCO.

Responsibilities

• Own the Azure Databricks landing zone architecture
• Define and implement Lakehouse patterns (bronze/silver/gold) using Delta Lake
• Establish and govern Unity Catalog
• Create platform governance standards
• Build CI/CD and release engineering for Databricks
• Implement observability and reliability practices
• Drive performance engineering
• Lead FinOps for Databricks
• Partner with Security & Compliance to meet enterprise and regulatory requirements
• Integrate the platform with Azure and enterprise services
• Enable MLOps and analytics at scale
• Coach, guide, and review solution designs from data engineers and analysts
• Develop and maintain platform documentation, onboarding materials, and runbooks
• Lead design and rollout of Databricks Mosaic AI
• Define prompt/response patterns, evaluation frameworks, guardrails, and cost controls
• Build LLMOps on the Lakehouse
• Demonstrate behaviors consistent with PEMCO's policies, values, code of ethics, and business conduct

Qualifications

• B.A./B.S. in Computer Science, Software/Systems Engineering, Data Engineering, Information Systems, or related field
• Master’s degree in a related field, such as Computer Science, Software Engineering, Artificial Intelligence, Machine Learning, Data Science or Data Engineering
• 6-8+ years designing or operating cloud data platforms at enterprise scale
• 3-4+ years hands on with Azure Databricks (or Databricks on cloud) including workspace administration, Unity Catalog, and Delta Lake
• 3+ years building secure data pipelines (batch/streaming) with Spark, SQL, Python/Scala, and Delta; experience with Auto Loader and DLT
• Proven experience in Databricks workspace and Lakehouse architecture
• 3+ years implementing CI/CD and infrastructure as code (Terraform strongly preferred) for data platforms
• Experience implementing lineage, cataloging, and governance (e.g., Purview/Unity Catalog) and integrating with enterprise SSO/SCIM
• Experience in the insurance domain, particularly Auto, Home, and Umbrella
• Databricks Certified Data Architecture Professional is preferred
• Familiarity with agile software delivery methodologies such as Scrum is preferred
• Familiarity with platforms and enabling tools such as Azure Machine Learning, Azure Databricks, Microsoft Fabric, Synapse Analytics, Power BI, Snowflake, and APIs like Azure OpenAI, Azure Cognitive Services, and Azure ML Endpoints is preferred
• Databricks Certified Data Architecture Professional is preferred
• Detail oriented with strong organizational, analytical, and communication skills
• Service minded with experience deciphering ambiguous requests and taking ownership
• Able to create and maintain positive relationships with employees at all levels of the organization
• Experience in the insurance industry and understanding of Auto, Home, and Umbrella insurance-related AI/ML applications
• Strong knowledge of real-time data streaming frameworks like Apache Kafka, Azure Event Hubs, and Delta Live Tables for Databricks
• Ability to work independently and manage tasks with minimal supervision
• Experience with unsupervised learning techniques to identify patterns and anomalies in data
• Strong knowledge of cloud-based security protocols, compliance standards, and governance frameworks is required
• Proficiency with AI/ML frameworks and embedding AI solutions into data ecosystems. Familiarity with AI/ML tools like TensorFlow, PyTorch, , Azure ML is required
• Proficiency in designing data models and architectures for both cloud and on-premises environments is required
• Team Player: Is responsive, flexible, and able to succeed in a team-oriented, collaborative environment, building effective working relationships and partnerships with internal partners, customers, and vendors
• Highly Analytical: Is passionate about working with disparate datasets bringing data together to answer business questions. Collaborates to create, manage, and translate data to meaningful insights. Can work with external vendors to integrate data for analysis; knows how to build efficient and scalable infrastructure and data models
• Problem Solver: Ability to analyze, diagnose and resolve complex unstructured problems quickly, efficiently, and collaboratively
• Communicator: The ability to communicate clearly and informatively, verbally and in writing, with colleagues, customers, and the community in both technical and non-technical professional language
• Job specific: Strong understanding of APIs, microservices architecture, and containerization (e.g., Kubernetes, Docker)
• Job specific: Skilled in generative AI models like GPT, DALL·E, etc.