Software Developer III

Key Responsibilities

• Lead migration of legacy SQL Server stored procedures and Azure Data Factory (ADF) pipelines to Databricks Lakehouse using Delta Lake architecture.
• Design scalable Lakehouse solutions utilizing Bronze, Silver, and Gold data layers.
• Build reusable ETL/ELT frameworks using PySpark, Delta Live Tables (DLT), and Databricks Workflows.
• Develop optimized dimensional models, star schemas, and Gold Layer datasets to maximize Microsoft Power BI performance.
• Optimize Databricks SQL Warehouses for DirectQuery and Import mode workloads.
• Implement advanced Databricks performance optimization techniques including Z-Ordering, Liquid Clustering, Data Skipping, Materialized Views, and file optimization.
• Create monitoring dashboards to analyze Databricks Unit (DBU) consumption and improve platform efficiency.
• Establish governance standards for cluster sizing, auto-scaling, serverless SQL compute, and cost optimization.
• Lead technical design sessions, architecture reviews, pair programming, mentoring, and code reviews.
• Produce technical documentation, architecture diagrams, migration playbooks, and operational standards.
• Drive knowledge transfer initiatives to ensure long-term operational success for internal engineering teams.
• Collaborate with business stakeholders, architects, developers, and analytics teams to deliver scalable data solutions.
• Communicate effectively with both technical and non-technical audiences while delivering projects on schedule.

Required Experience

• Bachelor's degree in Computer Science, Engineering, Information Systems, or a related technical discipline.
• 5+ years of Data Engineering, Data Platform Engineering, or Big Data development experience.
• 5+ years of experience working with cloud platforms such as Microsoft Azure, AWS, or Google Cloud Platform (GCP).
• Hands-on experience designing and implementing Databricks Lakehouse solutions in enterprise environments.
• Strong expertise with Apache Spark, PySpark, Delta Lake, Databricks SQL, Delta Live Tables (DLT), and Databricks Workflows.
• Experience migrating enterprise data warehouses from SQL Server or similar relational platforms to cloud-native analytics platforms.
• Experience building scalable ETL/ELT pipelines and metadata-driven ingestion frameworks.
• Strong knowledge of dimensional modeling, star schema design, enterprise data warehousing, and incremental data loading.
• Experience optimizing Power BI datasets and Databricks SQL Warehouses for enterprise reporting.
• Experience implementing Unity Catalog, Microsoft Entra ID (Azure AD), RBAC, row-level security, and column-level security.
• Strong SQL and Python programming skills.
• Experience leading technical projects, mentoring engineers, and conducting architecture reviews.
• Excellent analytical, troubleshooting, documentation, and communication skills.

Preferred Qualifications

• Experience with Azure Data Lake Storage (ADLS).
• Knowledge of distributed computing and cloud-native analytics platforms.
• Experience monitoring and optimizing Databricks cost and performance (DBU optimization).
• Familiarity with enterprise data quality frameworks and metadata-driven architecture.
• Microsoft Azure and/or Databricks certifications are highly desirable.