Databricks Engineer

About the role

The Databricks Engineer will design, develop, and optimize scalable data solutions on Databricks, leveraging PySpark or Scala for large-scale data processing. They will build and maintain ingestion pipelines, Declarative Pipelines (DLT), and Medallion Architecture (Bronze, Silver, Gold) to support enterprise analytics and reporting. They will develop robust data models and implement data quality, validation, and governance frameworks. They will create dynamic dashboards, Databricks Apps, and analytical solutions to deliver actionable business insights. They will optimize workloads, monitoring, and operational processes to ensure scalability, security, and cost efficiency.

Responsibilities

• Design, develop, and optimize scalable data solutions on Databricks, leveraging PySpark or Scala for large-scale data processing.
• Build and maintain ingestion pipelines, Declarative Pipelines (DLT), and Medallion Architecture (Bronze, Silver, Gold) to support enterprise analytics and reporting.
• Develop robust data models and implement data quality, validation, and governance frameworks.
• Create dynamic dashboards, Databricks Apps, and analytical solutions to deliver actionable business insights.
• Optimize workloads, monitoring, and operational processes to ensure scalability, security, and cost efficiency.

Requirements

• 8 years experience in IT, supporting the design, development, deployment, or delivery of technology solutions.
• 8 years experience with Databricks, including building and optimizing ETL/ELT data pipelines using Apache Spark.
• 8 years experience in data warehousing and dimensional data modeling (star/snowflake schemas).
• 8 years proficiency in SQL and Python (or Scala) for large-scale data processing.
• 8 years experience designing and developing dashboards and applications natively within Databricks (e.g., Databricks SQL dashboards, Databricks Apps).
• 8 years experience implementing data governance, data quality, and data security practices.
• 8 years experience implementing Lakeflow Declarative Pipelines (formerly Delta Live Tables/DLT) for building and managing production data pipelines.
• 8 years experience with Delta Lake, Medallion Architecture (Bronze, Silver, Gold layers), data lakehouse design, and creating and scheduling offline jobs using Lakeflow Jobs (formerly Databricks Workflows) or similar orchestration tools (e.g., Airflow).
• 8 years excellent communication skills, both verbal and written, including presenting insights to technical and business stakeholders.

Preferred Qualifications

• 1 year experience working in public sector or state government environments.
• 1 year Databricks certification (e.g., Databricks Certified Data Engineer Associate or Professional).
• 1 year experience with CI/CD practices for data pipelines (DevOps, Git-based workflows).