Big Data Architect Contractor

About the role

We are seeking a Big Data Architect Contractor to design, build, and optimize large-scale data solutions on Azure and Databricks. The role will focus on developing robust ETL/ELT pipelines, enabling high-performance data processing, and supporting end-to-end architecture across development, UAT, and production environments. The ideal candidate will work closely with client stakeholders and engineering teams to deliver scalable, secure, and cost-efficient data platforms.

Responsibilities

• Design, develop, and optimize ETL/ELT pipelines using Azure Data Factory (ADF) and Databricks
• Build and tune PySpark and Spark SQL notebooks for large-scale data transformation
• Support end-to-end data solutions across dev, UAT, and prod environments using Unity Catalog
• Participate in technical design discussions with client architects and cross-functional counterparts
• Collaborate with teams on data contracts and schema agreements
• Support the design and optimization of high-volume data pipelines
• Define and enforce engineering standards, including naming conventions, partitioning strategies, cluster configurations, and Spark tuning
• Drive performance optimization through AQE tuning, liquid clustering, broadcast joins, and shuffle partition management
• Support Databricks cluster policies, autoscaling configurations, and cost optimization strategies
• Perform root cause analysis on production incidents and implement permanent fixes
• Support junior and mid-level engineers through code reviews and pair programming
• Evaluate emerging technologies and recommend adoption where relevant, such as DABs, DLT, Auto Loader, Serverless Compute, and Event Hubs

Requirements

• Databricks — Advanced (6–9 years of experience)
• Strong hands-on experience with Azure Data Factory
• Advanced proficiency in Python, PySpark, and Spark SQL
• Strong background in big data architecture and scalable data platform design
• Experience with Unity Catalog and multi-environment delivery
• Knowledge of Spark performance tuning and optimization techniques
• Experience with cluster configuration, autoscaling, and cost management
• Strong analytical and troubleshooting skills for production support
• Ability to collaborate with technical teams and client stakeholders
• Experience supporting engineering standards and best practices

Pay

The expected pay range for this contract assignment is $65 - $70 per hour and will vary based on skills, experience, and location.

Schedule

Approximate hours per week: not specified (contract role).

Candidates must be geographically based in the United States and legally authorized to work in the United States without employer sponsorship.