Senior Databricks Data Engineer

About the role

Peak Performers has partnered with a Texas State Agency to assist in their search for a contract Sr. Databricks Data Engineer in Austin, Tx. In this position, you will be responsible for developing, maintaining, and optimizing big data solutions using the Databricks Unified Analytics Platform.

Responsibilities

• Designing and developing scalable data pipelines
• Implementing ETL/ELT workflows
• Optimizing Spark jobs
• Integrating with Azure Data Factory
• Automating deployments
• Collaborating with cross-functional teams
• Ensuring data quality, governance, and security
• Build AI accelerators and reusable components to boost development teams' productivity

Requirements

• Minimum 8 years of experience implementing ETL/ELT workflows for structured and unstructured data, designing and maintaining data models, schemas, and database structures, and ensuring data quality, validation, governance, security, and compliance
• 8 years of experience collaborating with cross-functional teams, including data scientists, analysts, stakeholders, and working in agile, multicultural environments
• 4 years of experience automating deployments with CI/CD tools and working with Azure cloud, DevOps, version control systems, and data storage solutions such as data lakes and data warehouses
• 4 years of experience with Python, R, SQL, troubleshooting, and debugging
• 3 years of experience designing and developing scalable data pipelines using Apache Spark on Databricks, including performance optimization, integration with Azure Data Factory, and use of Unity Catalog or Delta Lake
• 3 years of hands-on experience with Databricks notebooks, clusters, jobs, Delta Lake, and a strong understanding of Spark architecture, RDDs, DataFrames, and Spark SQL
• 2 years of experience building AI accelerators and reusable components, implementing GenAI/LLM applications and utilities, and supporting AI governance, observability, evaluation, vector stores, traceability, and SIEM integration
• 2 years of knowledge of ML libraries such as MLflow, Scikit-learn, and TensorFlow

Preferred

• Databricks Certified Associate Developer for Apache Spark
• Azure Data Engineer Associate certification

Pay & Schedule

• Rate: $86.00/hour W2 Contract
• Worksite Location: Austin, Tx 78751
• Hybrid
• 10 month contract with opportunity for extension