Senior Data Engineer

About the role

As a Senior Data Engineer/Lead Engineer, you will be responsible for designing, developing, and maintaining scalable Big Data solutions. You will work with large datasets, real‑time processing frameworks, and cloud‑based data platforms to enable data‑driven decision‑making for our clients.

Responsibilities

• Combine technical expertise and problem‑solving passion to work closely with clients, turning complex ideas into end‑to‑end data solutions that transform clients’ business.
• Translate client requirements into system design and develop solutions that deliver measurable business value.
• Lead, design, develop and deliver large‑scale data systems, data processing, data transformation, and data platform modernization initiatives.
• Build and optimise batch and streaming data pipelines across modern cloud data platforms and distributed processing frameworks.
• Support AI‑enabled engineering use cases by designing high‑quality data foundations, retrieval patterns, context engineering approaches, and scalable data services that power agentic and machine‑learning solutions.
• Automate data platform operations and manage post‑production systems, observability, quality, reliability, and operational processes, including telemetry pipelines that capture prompt, response, trace, latency, token, and cost data for AI‑enabled services in a queryable form.
• Conduct technical feasibility assessments and provide project estimates for the design and development of solutions.
• Mentor, support, and grow junior team members while contributing hands‑on to delivery.

Skills & Experience

• Demonstrable experience implementing end‑to‑end data pipelines and production‑grade data platforms.
• Hands‑on experience with at least one leading public cloud data platform: Amazon Web Services, Microsoft Azure, or Google Cloud Platform.
• Experience with Databricks as a data engineering platform (notebooks, jobs, Delta Lake, or similar lakehouse patterns) – strongly preferred.
• Strong Python proficiency and practical experience using Python‑based tooling for data engineering, automation, platform development, or AI engineering workflows.
• Implementation experience with column‑oriented databases (BigQuery, Redshift, Vertica, etc.), NoSQL databases (DynamoDB, Bigtable, Cosmos DB, etc.), and traditional relational databases (SQL Server, Oracle, MySQL).
• Experience building data pipelines for both streaming and batch integrations using tools/frameworks such as Glue ETL, Lambda, Google Cloud Dataflow, Azure Data Factory, Spark, Spark Streaming, or similar technologies.
• Experience with data modelling, warehouse design, fact/dimension implementations, and modern lakehouse or data mesh patterns.
• Familiarity with code repositories, continuous integration, automated testing, release management, and production support practices.
• Understanding of MLOps concepts and data‑engineering responsibilities required to support AI/ML deployment, validation, monitoring, rollback, and operational reliability.
• Ability to handle module or track‑level responsibilities while contributing hands‑on.
• Good communication skills and willingness to work as part of a collaborative, cross‑functional team.

AI Engineering & Modern Data Platform Experience

• Exposure to AI engineering patterns, including context engineering, retrieval‑augmented generation support patterns, agent architectures, and production data services that support AI‑enabled experiences.
• Experience building and maintaining pipelines behind retrieval systems (document parsing, chunking, metadata extraction, embedding generation, incremental re‑indexing) and working with vector databases, graph databases, semantic search, and knowledge retrieval structures.
• Exposure to agentic platforms or cloud AI services such as Vertex AI, Azure AI services, AWS AI services, or comparable platforms.
• Practical experience deploying agents, integrating agent frameworks, or supporting agentic workflows in production or near‑production environments (a plus).
• Experience building evaluation data infrastructure for AI systems, including