Associate Data Engineer - DataBricks

Principal Duties & Responsibilities

• Own the design and implementation of ETL workflows using Azure Data Factory and Databricks for compliance-related data ingestion and transformation.
• Integrate and normalize data from multiple structured and unstructured sources, ensuring adherence to strict data quality governance framework.
• Develop secure, auditable, and optimized pipelines for performance and reliability across sensitive financial datasets.
• Automate ingestion and transformation processes to enable real-time compliance analytics and reporting.
• Implement efficient storage formats and partitioning strategies for fast loading and retrieval.
• Utilize Databricks and Delta Lake for distributed processing and versioned data storage.
• Enforce data retention, archiving, and purging policies aligned with global regulatory requirements.
• Migrate legacy application logic into modern Azure Databricks/Data Lake and SQL environments.
• Maintain comprehensive data lineage, metadata, and audit trails using Azure Purview or custom frameworks.
• Collaborate with governance teams and compliance analysts to define requirements and manage access controls.
• Participate in Agile development cycles, sprint planning, and compliance reviews.
• Review code changes and support production deployments.

Job Specifications

• 3+ years of experience in data engineering within financial services, preferably compliance/regulatory.
• Advanced proficiency in SQL, Python, and Azure Databricks.
• Hands-on experience with Azure Data Factory, Databricks notebooks, and ADLS.
• Expertise in data wrangling, modeling, and analytics enablement for compliance use cases.
• Experience integrating Microsoft ADF and Databricks for complex ETL development.
• Familiarity with data governance, quality frameworks, and audit processes.
• Strong communication and collaboration skills.
• Experience with Git/GitLab and CI/CD tools such as Azure DevOps.
• PREFERRED: Certifications in Azure Data Engineering, Databricks, or Financial Compliance Technology.
• PREFERRED: Bachelor’s or Master’s in Computer Science, Data Engineering, or related field.