GCP Data Engineer

About the role

This is primarily a GCP Data Engineer / Python Engineer role with strong experience in building data pipelines, cloud-based applications, and AI-enabled solutions.

Responsibilities

• Develop and maintain scalable, efficient, and reliable data pipelines using Google Cloud Platform (GCP) services such as BigQuery, Cloud Storage, Cloud Functions, Cloud Run, Dataflow, Cloud Composer (Airflow), Pub/Sub.
• Design and implement real-time data processing systems using event-driven architectures and streaming data pipelines.
• Optimize data warehousing and modeling processes including schema design, partitioning, clustering, and materialized views.
• Build and manage containerized applications using Docker and Kubernetes.
• Create and integrate REST APIs and other external systems using Swagger/OpenAPI.
• Implement continuous integration and continuous deployment (CI/CD) pipelines using tools like Git, GitHub Actions, Jenkins, and Cloud Build.
• Set up and configure monitoring and logging systems including Cloud Monitoring, Cloud Logging, and Error Reporting.
• Ensure security and compliance with Identity and Access Management (IAM), access controls, data governance, and automated CVE remediation.
• Utilize AI/ML enabled services from Google Cloud Platform (GCP) such as Vertex AI and Gemini to enhance data processing and analysis.

Requirements

• Strong proficiency in Python with a solid understanding of object-oriented programming (OOP) concepts, modular and reusable code, and clean coding practices.
• Extensive experience with Google Cloud Platform (GCP) services including BigQuery, Cloud Storage, Cloud Functions, Cloud Run, Dataflow, Cloud Composer (Airflow), Pub/Sub.
• Proven track record in designing and implementing ETL/ELT pipelines and streaming data pipelines.
• Experience with Docker and Kubernetes for containerization and orchestration.
• Knowledge of REST APIs and their integration with external systems.
• Experience with CI/CD practices and tools such as Git, GitHub Actions, Jenkins, and Cloud Build.
• Understanding of monitoring and logging tools like Cloud Monitoring, Cloud Logging, and Error Reporting.
• Experience with security best practices including IAM, access management, data governance, and automated CVE remediation.
• Familiarity with AI/ML services from Google Cloud Platform (GCP) such as Vertex AI and Gemini.

Qualifications

• Advanced degree in Computer Science, Information Technology, or related field.
• Minimum 5 years of relevant work experience in data engineering, software development, or a similar field.
• Experience with Jira for project management and Agile/SAFe delivery practices.

Skills

• Python programming (OOP concepts, reusable code, clean coding practices)
• Google Cloud Platform (GCP) - BigQuery, Cloud Storage, Cloud Functions, Cloud Run
• Data Engineering - ETL/ELT pipelines, Dataflow, Cloud Composer (Airflow), Pub/Sub
• Real-Time Data Processing - Event-driven architecture, Streaming data pipelines
• BigQuery/Data Warehouse - Schema design, partitioning, clustering, performance optimization
• Docker & Kubernetes
• REST APIs & integrations
• CI/CD - Git, Cloud Build, GitHub Actions, Jenkins
• Miscellaneous - Monitoring & Logging, Security, IAM, access management, data governance, CVE remediation
• Vertex AI / Gemini
• GitHub Copilot / AI-assisted development
• Integration & Functional Testing
• Agile / SAFe

Benefits

N/A

Pay

N/A

Schedule

N/A