Databricks Consultant

Location: Houston, MS — Hybrid (2-3 days)
Duration: 6 months

About the role

Design, develop, and implement scalable data solutions using the Databricks platform. Apply strong expertise in data engineering, cloud platforms, Apache Spark, and modern data architectures. Collaborate closely with business and technical stakeholders to understand requirements and deliver effective solutions.

Responsibilities

• Design and develop data pipelines using Databricks, Apache Spark, and PySpark.
• Build and maintain ETL/ELT processes for large-scale data ingestion and transformation.
• Implement Data Lake and Lakehouse architectures on Databricks.
• Integrate data from multiple sources, including databases, APIs, and cloud storage.
• Optimize Spark jobs and Databricks workloads for performance, scalability, and cost efficiency.
• Collaborate with business users, solution architects, developers, and technical stakeholders.
• Develop data models to support reporting, analytics, and AI/ML initiatives.
• Implement standards for data quality, data governance, data security, and data monitoring.
• Support production deployments, perform troubleshooting, and tune performance.
• Prepare technical documentation.
• Provide knowledge transfer and technical guidance to project teams.

Requirements

• 7+ years of experience in data engineering or related roles.
• Strong hands-on experience with Databricks.
• Expert-level knowledge of Apache Spark, PySpark, and SQL.
• Experience with cloud platforms such as Azure, AWS, or GCP.
• Strong knowledge of Delta Lake, Unity Catalog, and Databricks Workflows.
• Experience with data warehousing, data lakes, and Lakehouse architecture.
• Strong understanding of ETL/ELT frameworks and data modeling.
• Familiarity with CI/CD, Git, and DevOps practices.
• Excellent problem-solving and communication skills.

Qualifications

• Bachelor’s degree in Computer Science, Information Technology, Engineering, or a related technical field.
• 5+ years of experience in data engineering or big data roles.
• 2+ years of hands-on Databricks implementation experience.

Preferred Skills

• Experience with Azure Data Factory (ADF), Azure Synapse, or AWS Glue.
• Knowledge of machine learning on Databricks and MLOps.
• Experience with streaming technologies such as Apache Kafka and Structured Streaming.
• Databricks or cloud platform certifications.
• Databricks Certified Data Engineer Associate/Professional certification.
• Experience in industry domains such as utilities, energy, banking, healthcare, or retail.
• Experience with Agile delivery methodologies.

Primary Technologies

• Databricks
• PySpark
• Spark SQL
• Delta Lake
• Azure, AWS, GCP
• Azure Data Factory (ADF)
• Apache Kafka
• Git
• CI/CD