New contract opening of Python Pyspark Databricks -- Boston, MA

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Python, PySpark, and Databricks.
• Build and optimize ETL/ELT workflows to ingest, transform, and process structured and unstructured data.
• Develop high-performance Spark applications for large-scale data processing.
• Implement data quality checks, validation frameworks, and error-handling mechanisms.
• Work with Delta Lake to build reliable and efficient data lakes.
• Optimize Spark jobs for performance, scalability, and cost efficiency.
• Collaborate with data architects, data scientists, analysts, and business stakeholders to understand data requirements.
• Integrate data from multiple sources including databases, APIs, cloud storage, and streaming platforms.
• Participate in code reviews, testing, deployment, and production support.
• Monitor data pipelines and troubleshoot performance issues.
• Follow data governance, security, and best practices throughout the development lifecycle.
• Create technical documentation and maintain data pipeline documentation.

Required Skills

• Strong experience with Python programming.
• Hands-on experience with PySpark for distributed data processing.
• Extensive experience working with Databricks.
• Strong understanding of Apache Spark architecture and optimization techniques.
• Experience with Delta Lake.
• Knowledge of SQL and relational databases.
• Experience designing and maintaining ETL/ELT pipelines.
• Familiarity with data modeling and data warehousing concepts.
• Experience with Git and version control.
• Understanding of CI/CD pipelines and DevOps practices.

Pay

N/A

Schedule

N/A

Benefits

N/A