Lead Consultant | Cloud Platform | Google Cloud - Architecture

About the role

We are seeking a Python-based platform engineer to design and build a containerized API layer that abstracts and governs interactions with Apache Spark through a well-defined API contract. This role focuses on building platform capabilities, not simply consuming existing data tools—enabling consistent, secure, and scalable access to Spark-based data pipelines.

Responsibilities

• Design and implement a containerized API layer for Apache Spark interactions
• Ensure consistent, secure, and scalable access to Spark-based data pipelines
• Manage complex software application development and support projects
• Coordinate delivery and manage offshore resources
• Collaborate with onsite team members and client stakeholders

Requirements

• At least 10 years of experience in the IT industry
• 8+ years of hands-on Python development experience
• Good hands-on experience with GCP skills
• Experience building control planes and applying design patterns
• Ability to manage offshore resources and coordinate with clients

Skills

• Python, PySpark
• FastAPI, ARQ, Kubernetes, Pytest
• Modular containerized services
• Asynchronous processing
• Job-based work queue execution
• Immutability and idempotency
• Metadata management
• Policy enforcement
• Semantic modeling
• Observability and telemetry
• Side effect capability injection
• Functional and behavioral testing

Pay

$52.70 - $54.92/hr