Systems Analyst III / Databricks Data Engineer

About the role

Texas Health and Human Services Commission (HHSC) is seeking an experienced Systems Analyst III / Databricks Data Engineer to design, develop, and optimize enterprise-scale data solutions using the Databricks platform. The ideal candidate will have extensive experience building scalable ETL/ELT pipelines, implementing modern Lakehouse architecture, and developing analytical dashboards that enable business decision-making.

Responsibilities

• Design, develop, and maintain scalable enterprise data solutions using the Databricks platform.
• Develop and optimize ETL/ELT pipelines using Apache Spark (PySpark or Scala).
• Build and maintain Lakehouse architectures utilizing Delta Lake and Medallion Architecture (Bronze, Silver, Gold).
• Implement Lakeflow Declarative Pipelines (formerly Delta Live Tables) for production-grade data processing.
• Develop and schedule workflows using Lakeflow Jobs (formerly Databricks Workflows) or equivalent orchestration tools such as Apache Airflow.
• Design and maintain dimensional data models including Star and Snowflake schemas.
• Develop high-performance SQL queries for analytical workloads.
• Create interactive dashboards and applications using Databricks SQL Dashboards and Databricks Apps.
• Implement enterprise data governance, metadata management, data quality validation, and security best practices.
• Monitor pipeline performance and optimize workloads for scalability, reliability, and cost efficiency.
• Analyze business requirements and translate them into technical solutions.
• Perform system analysis, troubleshooting, and production support activities.
• Collaborate with business users, project managers, developers, and architects throughout the software development lifecycle.
• Document technical designs, workflows, data models, and operational procedures.
• Present technical findings and business insights to both technical and non-technical stakeholders.

Qualifications

Bachelor's degree in Computer Science, Information Systems, Engineering, or a related field (or equivalent experience). Minimum 8 years of IT experience supporting the design, development, deployment, or delivery of technology solutions. Minimum 8 years of experience with Databricks.

Skills

• Strong expertise in Apache Spark using PySpark or Scala.
• Extensive experience building enterprise ETL/ELT pipelines.
• Strong knowledge of Delta Lake.
• Experience implementing Lakeflow Declarative Pipelines (Delta Live Tables).
• Experience with Medallion Architecture (Bronze, Silver, Gold).
• Expertise in Lakehouse architecture.
• Strong experience with SQL.
• Strong programming skills in Python (preferred) or Scala.
• Experience designing dimensional data models (Star/Snowflake schema).
• Experience developing Databricks SQL Dashboards and Databricks Apps.
• Experience implementing data governance, data quality, security, and validation frameworks.
• Experience scheduling and orchestrating jobs using Lakeflow Jobs or Apache Airflow.
• Excellent verbal and written communication skills.
• Ability to work independently while collaborating across cross-functional teams.

Technical Skills

• Data Engineering: Databricks, Apache Spark, PySpark, Scala, Delta Lake, Delta Live Tables (DLT), Lakeflow Declarative Pipelines, Lakeflow Jobs, Databricks Workflows, Data Lakehouse, Medallion Architecture
• Programming: Python, SQL, Scala
• Data Warehousing: Dimensional Modeling, Star Schema, Snowflake Schema, ETL, ELT, Reporting & Analytics: Databricks SQL, Databricks Dashboards, Databricks Apps
• Data Management: Data Governance, Data Quality, Data Validation, Data Security, Metadata Management
• DevOps & Tools: Git, CI/CD, Azure DevOps (preferred), Apache Airflow

Soft Skills

• Strong analytical and problem-solving skills
• Excellent communication and presentation abilities
• Ability to interact with business and technical stakeholders
• Strong documentation skills
• Self-motivated and detail-oriented
• Ability to manage multiple priorities in a fast-paced environment