GCP Data Engineer

Responsibilities

• Write efficient, optimized SQL queries to extract, clean, transform, and analyze large and complex datasets from multiple sources.
• Automate repetitive data workflows and reporting processes to improve efficiency and reduce manual effort.
• Document methodologies, code, and findings clearly for reproducibility and knowledge sharing.
• Execute migration tasks under the guidance of senior engineers/architects.
• Build ingestion scripts, transformation logic, and end-to-end data pipelines.
• Support migration of data from Teradata to BigQuery.
• Implement pipeline monitoring, logging, and alerting to ensure reliability.
• Support testing, validation, and reconciliation of migrated datasets to ensure data integrity.
• Maintain and update technical documentation and operational runbooks.
• Collaborate with cross-functional teams including data analysts, architects, and business stakeholders to understand requirements.

Requirements

• 6–10 years of experience.
• Strong SQL and Python skills.
• Basic-to-intermediate experience with GCP services (BigQuery, GCS, Composer, Cloud Functions).
• Solid understanding of ETL/ELT data flows and pipeline design.
• Experience working with APIs, file-based ingestion, and orchestration tools.
• Ability to test, validate, and debug data pipelines effectively.
• Hands-on exposure to migration tools such as Storage Transfer Service, gsutil, and bq CLI.
• Working knowledge of Spark (Dataproc) is a plus.
• Understanding of Teradata-to-GCP mapping concepts (e.g., Teradata → GCS, ADF → Composer).
• Familiarity with CI/CD concepts and practices.
• Exposure to version control systems such as Git for code management and collaboration.
• Understanding of data security, access controls, and governance best practices in cloud environments.
• Familiarity with data quality frameworks and reconciliation techniques.
• Good communication and problem-solving skills, with the ability to work independently and in a team setting.