Digital Platforms, Senior Data Engineer

About the Role

The Senior Data Engineer (Databricks Platform) is a senior individual contributor responsible for the hands-on design, development, optimization, and operational support of the enterprise data platform built on Databricks and AWS technologies. This role ensures scalable, secure, and high-performance data pipelines, models, and integrations that power analytics, reporting, and advanced AI/ML use cases across the organization.

Responsibilities

• Data Platform Engineering
  • Design, develop, and maintain scalable data pipelines using Databricks (Spark / Delta Lake).
  • Build robust ETL/ELT workflows leveraging PySpark and Databricks SQL.
  • Implement and support medallion architecture (Bronze, Silver, Gold layers).
  • Design and maintain efficient data models, including 3NF, star schema, and canonical models, as appropriate.
  • Support and maintain Data Catalog and Data Governance frameworks to ensure data quality and compliance.
  • Build and maintain CI/CD pipelines to enable reliable deployment across environments.
  • Collaborate with external partners for development activities and ongoing production support.
• Cloud & AWS Environment Support
  • Demonstrate hands-on experience with AWS Glue, S3, and Lambda.
  • Provide operational support for AWS-hosted infrastructure, including application services, data services, and system integrations.
  • Monitor cloud environments for availability, performance, security, and cost efficiency; escalate risks and issues to the Technical Lead when required.
  • Support deployment activities and ensure environment readiness in collaboration with DevOps and technical stakeholders.
  • Contribute to cloud operational best practices focused on scalability, resilience, security, and incident response.
• Testing Support
  • Partner with Testing and QA teams to support User Acceptance Testing (UAT) and validation activities by:
    • Reviewing test cases for technical accuracy and risk coverage
    • Triaging defects identified during testing cycles
    • Ensuring test environment readiness and stability
  • Provide technical input during release readiness discussions and risk evaluations.
• Cross-Functional Collaboration
  • Collaborate closely with Product Managers, Project Managers, QA, DevOps, and external development partners to ensure successful solution delivery and ongoing platform stability.
  • Translate business and operational requirements into scalable technical solutions aligned with approved architecture and enterprise standards.
  • Support secure and reliable integration with internal enterprise systems and data sources.
• Compliance & Operational Expectations
  • Ensure adherence to all applicable PCI, cGMP, GCP, and regulatory policies.
  • Participate in occasional overtime and/or weekend work as required to support critical releases or production needs.
  • Maintain consistent attendance and reliable availability as essential functions of the role.
  • Perform additional duties as assigned by the Data Platform Technical Lead or management.

Requirements

• Deep expertise in distributed data processing, Delta Lake architecture, data modeling, ETL/ELT frameworks, and cloud environments (AWS).
• Focus on execution excellence, platform reliability, and technical problem resolution.