Senior Databricks Data Engineer (Python, Spark, SQL, Delta Lake, Lakeflow, ETL/ELT) NEW!

Location: Austin, TX

About the role

Design, develop, and maintain scalable ETL/ELT data pipelines using Databricks and Apache Spark. Build and optimize enterprise data solutions utilizing Delta Lake and Lakehouse Architecture.

Responsibilities

• Design and implement Bronze, Silver, and Gold (Medallion) data layers.
• Develop and maintain Lakeflow Declarative Pipelines (formerly Delta Live Tables - DLT) for production-grade data processing.
• Create, schedule, and manage workflows using Lakeflow Jobs (formerly Databricks Workflows) or comparable orchestration tools such as Apache Airflow.
• Design and maintain enterprise data warehouse solutions using dimensional modeling techniques, including Star Schema and Snowflake Schema.
• Develop high-performance SQL queries and optimize Spark jobs for large-scale data processing.
• Build dashboards and applications using Databricks SQL Dashboards and Databricks Apps.
• Implement enterprise data governance, data quality, data security, and metadata management best practices.
• Monitor and optimize data pipeline performance, scalability, and reliability.
• Implement CI/CD pipelines for data engineering solutions using Git-based workflows and DevOps practices.
• Collaborate with architects, business analysts, and stakeholders to understand business requirements and translate them into technical solutions.
• Troubleshoot production issues and provide ongoing support for enterprise data platforms.
• Document architecture, data flows, pipeline designs, and operational procedures.
• Present technical solutions and project updates to both technical and business stakeholders.
• Follow software development lifecycle (SDLC) and Agile development methodologies.

Requirements

• Strong experience supporting the design, development, deployment, and delivery of enterprise technology solutions.
• Extensive experience with Databricks and Apache Spark.
• Experience building and optimizing ETL/ELT data pipelines.
• Strong experience with Delta Lake and Lakehouse Architecture.
• Experience implementing Lakeflow Declarative Pipelines (Delta Live Tables/DLT).
• Experience creating and scheduling production jobs using Lakeflow Jobs (Databricks Workflows) or similar orchestration tools.
• Strong knowledge of SQL and Python (Scala experience is a plus).
• Experience with data warehousing and dimensional data modeling (Star and Snowflake schemas).
• Experience designing Databricks SQL Dashboards and Databricks Apps.
• Strong understanding of data governance, data quality, data security, and compliance best practices.
• Experience implementing CI/CD processes for data engineering projects using Git-based workflows.
• Excellent troubleshooting, analytical, and problem-solving skills.
• Strong verbal and written communication skills.
• Ability to collaborate effectively with cross-functional teams.

Preferred Qualifications

• Experience working in public sector or state government environments.
• Databricks Certified Data Engineer Associate or Professional Certification.
• Experience with Azure Databricks, AWS, or Google Cloud Platform (GCP).
• Experience using orchestration tools such as Apache Airflow.
• Experience with Agile/Scrum development methodologies.
• Familiarity with enterprise data governance and regulatory compliance standards.