Databricks Engineer

Allied Consultants, Inc is an Austin-based firm with over 34 years of experience delivering top-tier technical and business professionals within Texas State Agencies.

About the role

We are seeking an experienced Databricks Engineer to play a key role within a high-impact technical services team. This position is hybrid, requiring 2 days onsite (Mondays and Thursdays) with 3 days remote.

Responsibilities

• Design, develop, and optimize scalable data solutions on Databricks, leveraging PySpark or Scala for large-scale data processing.
• Build and maintain ingestion pipelines, Declarative Pipelines (DLT), and Medallion Architecture (Bronze, Silver, Gold) to support enterprise analytics and reporting.
• Develop robust data models and implement data quality, validation, and governance frameworks.
• Create dynamic dashboards, Databricks Apps, and analytical solutions to deliver actionable business insights.
• Optimize workloads, monitoring, and operational processes to ensure scalability, security, and cost efficiency.
• Analyze user requirements, procedures, and problems to automate processing or improve existing systems.
• Confer with personnel to analyze current operational procedures, identify problems, and determine input/output requirements.
• Write detailed descriptions of user needs, program functions, and steps required to develop or modify computer programs.
• Review system capabilities, specifications, and scheduling limitations to assess feasibility of requested changes.

Requirements

• 8+ years of experience in IT, supporting the design, development, deployment, or delivery of technology solutions.
• 8+ years of experience with Databricks, including building and optimizing ETL/ELT data pipelines using Apache Spark.
• 8+ years of experience in data warehousing and dimensional data modeling (star/snowflake schemas).
• 8+ years of proficiency in SQL and Python (or Scala) for large-scale data processing.
• 8+ years of experience designing and developing dashboards and applications natively within Databricks (e.g., Databricks SQL dashboards, Databricks Apps).
• 8+ years of experience implementing data governance, data quality, and data security practices.
• 8+ years of experience implementing Lakeflow Declarative Pipelines (formerly Delta Live Tables/DLT) for building and managing production data pipelines.
• 8+ years of experience with Delta Lake, medallion architecture (bronze/silver/gold layers), data lakehouse design, and creating and scheduling offline jobs using Lakeflow Jobs (formerly Databricks Workflows) or similar orchestration tools (e.g., Airflow).
• Excellent communication skills, both verbal and written, including presenting insights to technical and business stakeholders.

Qualifications

• 1+ year of experience working in public sector or state government environments (preferred).
• Databricks certification (e.g., Databricks Certified Data Engineer Associate/Professional) (preferred).
• Experience with CI/CD practices for data pipelines (DevOps, Git-based workflows) (preferred).

Benefits

• Highly competitive pay rates.
• Medical insurance with employer cost sharing.
• Life insurance.
• 401(K) plan with company match.
• Flexible spending through a cafeteria plan.

Candidates selected for interviews will be subject to a criminal background check and may be required to pass a drug screening. All offers of employment are contingent upon successful completion of these checks.