Senior Cloud Data Engineer
InvestM Technology LLC · San Francisco, CA · 2 wk ago
On-siteInformation TechnologyContract
Responsibilities
- Design and implement scalable cloud data warehouse architectures, defining layer structures, schema design patterns, and partitioning/clustering strategies.
- Architect physical and logical data models that balance query performance, storage efficiency, and business domain clarity, applying dimensional modeling and data techniques where appropriate.
- Adopt cloud data frameworks to build modular, reusable SQL models across staging, intermediate, and mart layers with full documentation and lineage.
- Build and maintain data pipelines that ingest data from diverse sources, Snowflake shares, databases, APIs, event streams, and flat files into the cloud data warehouse reliably and at scale.
- Implement batch and near-real-time ingestion patterns using cloud-native tools, managing incremental loads, CDC (change data capture), and idempotent pipeline design.
- Optimize query performance through materialization strategies, clustering keys, and warehouse-specific query tuning techniques.
- Implement and maintain role-based access control (RBAC), column-level security, dynamic data masking, and row-level access policies to enforce least-privilege and data privacy requirements.
- Establish and maintain CI/CD pipelines for warehouse deployments automating testing, and promotion of transformation code across dev, UAT, and production environments.
- Operate within an Agile environment using JIRA to manage work items, participate in sprint planning, and deliver high-quality solutions on a consistent cadence.
- Apply AI-assisted development tools pragmatically across the engineering lifecycle, accelerating warehouse transformation authoring, data quality automation, and documentation workflows.
Requirements
- 10+ years of data engineering experience, with a proven track record of hands-on development and end-to-end solution delivery.
- Proven ability to design scalable cloud data warehouse architectures, defining layered structures, schema design patterns, and physical data models that balance query performance, storage efficiency, and business domain clarity.
- Deep expertise in Snowflake, including data modeling, performance tuning, and cost-efficient design, along with experience managing vendor data shares for secure and governed access.
- Advanced SQL skills with a strong foundation in data warehousing concepts, including dimensional modeling, incremental processing, slowly changing dimensions, and semantic layers.
- Hands-on experience designing and operating cloud data solutions on Azure including ADF, ADLS Storage, with a strong grasp of cloud-native ingestion patterns and pipeline orchestration.
- Proficiency in modern data transformation frameworks (dbt) including modular model design across layered warehouse architecture.
- Familiarity with streaming and near-real-time ingestion patterns (e.g., Azure Event Hubs, Kafka) including incremental load design, CDC, and latency-aware pipeline considerations.
- Strong understanding of platform reliability engineering for data warehouse covering orchestration, backfill and reprocessing strategies, and warehouse performance optimization.
- Experience designing and maintaining data quality frameworks with operational alerting and runbooks to support SLA-driven reliability.
- Experience implementing CI/CD pipelines for dbt and Snowflake workloads, including Git-based workflows, automated testing, and environment promotions.
- Strong written and verbal communication skills with the ability to produce clear data model documentation, pipeline runbooks, and data dictionaries.
- Bachelor's degree in computer science, information systems, or a related field.
- Experience in asset management, financial services, or investment-related, preferred.