PySpark Consultant

Responsibilities

• Develop and maintain complex data processing and analytical solutions using Python/Pyspark, including advanced scripting and framework development.
• Architect, design, implement, and manage solutions within distributed data processing platforms, specifically with the Hadoop ecosystem (preferably Cloudera distributions).
• Leverage key big data components such as distributed file systems (e.g., HDFS), data warehousing solutions (e.g., Hive), data transformation frameworks (e.g., Pig), and data ingestion tools (e.g., Sqoop), alongside hands-on experience with NoSQL databases (preferably MongoDB).
• Architect, design, and implement highly scalable data pipelines, leveraging industry-standard ETL tools and frameworks for efficient data extraction, transformation, and loading into various relational databases and data warehouses.
• Strategically plan and implement migrations to cloud-native, serverless ETL solutions.
• Understand and apply advanced data modeling principles and practical experience in data warehouse design and development, ensuring data integrity, scalability, security, and optimal performance.
• Leverage advanced AI tools, such as Devin, for efficient code refactoring, optimization, and identifying potential code improvements, thereby enhancing code quality and developer productivity.
• Act as a development lead, focusing on inculcating best practices, guiding junior developers, and taking the initiative to reengineer and re-architect generic frameworks to enhance efficiency and scalability.
• Manage and implement successful projects, demonstrating subject matter expertise in at least one area of Applications Development.
• Adjust priorities quickly as circumstances dictate.
• Demonstrate leadership and project management skills.
• Consistently demonstrate clear and concise written and verbal communication.

Requirements

• 10+ years of relevant experience in Apps Development or systems analysis role.
• Extensive Python and/or big data expertise, including mastery of Python/Pyspark.
• Extensive experience (8+ years) architecting, designing, implementing, and managing solutions within distributed data processing platforms, specifically with the Hadoop ecosystem (preferably Cloudera distributions).
• Proficient in leveraging key big data components such as distributed file systems (e.g., HDFS), data warehousing solutions (e.g., Hive), data transformation frameworks (e.g., Pig), and data ingestion tools (e.g., Sqoop), alongside hands-on experience with NoSQL databases (preferably MongoDB).
• Proven ability to architect, design, and implement highly scalable data pipelines, leveraging industry-standard ETL tools and frameworks for efficient data extraction, transformation, and loading into various relational databases and data warehouses.
• Understanding of advanced data modeling principles and practical experience in data warehouse design and development, ensuring data integrity, scalability, security, and optimal performance.
• Ability to leverage advanced AI tools, such as Devin, for efficient code refactoring, optimization, and identifying potential code improvements, thereby enhancing code quality and developer productivity.
• Ability to act as a development lead, focusing on inculcating best practices, guiding junior developers, and taking the initiative to reengineer and re-architect generic frameworks to enhance efficiency and scalability.
• Experience managing and implementing successful projects, demonstrating subject matter expertise in at least one area of Applications Development.
• Ability to adjust priorities quickly as circumstances dictate.
• Leadership and project management skills.
• Clear and concise written and verbal communication skills.