Lead PySpark Developer (7234-1)
AleraInfoTech · Owings Mills, MD · 2 mo ago
Information Technology$68/hrFull-time
Location: Owings Mills, MD | Type: Contract | Work Mode: On site | Max pay rate: $68/hr | Experience level: Mid-senior
Requirements
- 7+ years of experience in Amazon Web Service (AWS) Cloud Computing
- 10+ years of experience in big data and distributed computing
- Very strong hands-on experience with PySpark, Apache Spark, and Python
- Strong hands-on experience with SQL and NoSQL databases (DB2, PostgreSQL, Snowflake, etc.)
- Proficiency in data modeling and ETL workflows
- Proficiency with workflow schedulers like Airflow
- Hands-on experience with AWS cloud-based data platforms
- Experience in DevOps, CI/CD pipelines, and containerization (Docker, Kubernetes) is a plus
- Strong problem-solving skills and ability to lead a team
- Experience with DBT and AWS Astronomer
Responsibilities
- Lead the design, development, and deployment of PySpark-based big data solutions
- Architect and optimize ETL pipelines for structured and unstructured data
- Collaborate with client, data engineers, data scientists, and business teams to understand requirements and provide scalable solutions
- Optimize Spark performance through partitioning, caching, and tuning
- Implement best practices in data engineering (CI/CD, version control, unit testing)
- Work with cloud platforms like AWS
- Ensure data security, governance, and compliance
- Mentor junior developers and review code for best practices and efficiency