DirectJobHub
Jobs · Engineering

Sr Data Software Engineer - Remote - W2 Only

Saransh Inc · United States · 1 mo ago
RemoteRemoteEngineeringContract

About the Role

We are seeking a Senior Data Software Engineer for a remote W2 position (EST) to join a client project focused on modernizing their data platform. The role involves leading a large-scale migration from Delta Lake to Apache Iceberg while ensuring fine-grained access control (FGAC) policies are correctly preserved. This position requires deep platform fluency across the modern data lake stack and hands-on experience with AWS services.

Project Overview

The client has a central data platform built on AWS Glue Data Catalog with a developed FGAC system being adopted across data-producing teams. As part of this adoption, application teams must review their data pipelines to identify queries affected by new access controls and modify them accordingly. The role centers on two main initiatives: the Delta-to-Iceberg migration framework and FGAC preservation during migration.

Delta-to-Iceberg Migration Responsibilities

  • Design and build a migration feature/framework to convert tables from Delta Lake format to Apache Iceberg at scale
  • Leverage deep understanding of Iceberg internals (metadata layers, manifests, snapshots, partitioning, schema evolution) to ensure correct and efficient migration
  • Validate data integrity, schema fidelity, and partitioning strategy parity between source Delta tables and migrated Iceberg tables
  • Optimize migration processes for performance and reliability across large volumes of tables and data
  • Identify and resolve edge cases, compatibility issues, or performance bottlenecks during migration at scale

FGAC (Fine-Grained Access Control) Responsibilities

  • Ensure Row-Level Security (RLS) and Column-Level Security (CLS) policies are correctly preserved or re-implemented on migrated Iceberg tables
  • Integrate migration workflows with AWS Lake Formation to maintain governance and access control continuity
  • Collaborate with security/governance teams to validate that FGAC policies enforce correctly post-migration
  • Support GDC (Global Data Catalog / Governed Data Catalog) integration to ensure migrated tables remain properly cataloged and governed
  • Build validation/reconciliation tooling to confirm access control parity between pre- and post-migration states

General Responsibilities

  • Document migration architecture, access control mapping logic, and operational runbooks
  • Collaborate with data platform, governance, and security teams throughout the migration lifecycle
  • Monitor and troubleshoot production issues related to migrated tables and access enforcement
  • Contribute to best practices and reusable tooling for future Delta-to-Iceberg migration phases

Required Qualifications

  • Strong, hands-on knowledge of Apache Iceberg internals (table format, metadata management, snapshots, partitioning, schema evolution)
  • Proven experience performing Delta-to-Iceberg migrations at scale
  • Experience with AWS Lake Formation for access control and data governance
  • Practical experience implementing or preserving Row-Level Security (RLS) and Column-Level Security (CLS)
  • Experience with GDC (Global/Governed Data Catalog) or equivalent data cataloging/governance systems
  • Solid understanding of distributed data processing and large-scale data migration challenges
  • Strong SQL skills and experience with table format internals (Delta Lake, Iceberg, or similar)

Preferred Qualifications

  • Experience with Spark for large-scale data processing and migration tooling
  • Familiarity with AWS data platform services (S3, Glue, Athena)
  • Experience with data governance frameworks and compliance-driven access control requirements
  • Prior experience building reusable migration frameworks or tooling for table format conversions
  • Scripting/automation experience (Python, Scala) for migration validation and tooling

Similar jobs