Data Engineer

About the role

Data Engineer role is responsible for design, build, test, deployment, and support of enterprise data pipelines (CI/CD), lakehouse tables, transformation frameworks, change data capture (CDC), metadata controls, data quality processes, and governed data products. This role implements reliable data movement and transformation patterns across modern cloud data platforms, including Microsoft Fabric, Azure data services, Databricks, Snowflake, or comparable lakehouse and warehouse technologies. The Data Engineer transforms data using PySpark, Spark SQL, Python, and SQL, with deep understanding of related frameworks.

The position requires practical understanding of medallion architecture, Delta / open table formats, data catalogs, data products, dimensional modeling, metadata management, master data management, governance, security, data mesh, domain-driven design, DevOps, observability, and operational support. The engineer supports the full lifecycle of data products: ingestion, raw/bronze landing, source registration, technical staging, silver conformance, gold semantic publishing, metadata capture, quality validation, promotion, monitoring, and support. The engineer partners with analysts, architects, business SMEs, data stewards, security teams, and platform administrators to deliver data products that are trusted, documented, secure, reusable, and performant.

Responsibilities

• Design, build, test, deploy, and support modern data platform pipelines and lakehouse or warehouse data products.
• Implement ingestion, staging, transformation, conformance, and publishing patterns across raw/bronze, stage, silver, gold, and certified consumption layers.
• Build PySpark, Spark SQL, Python, and SQL transformations that are distributed, restartable, idempotent, and operationally supportable.
• Implement full-refresh, incremental, CDC, merge, and partition-aware load patterns where appropriate.
• Design source registration, audit stamping, row hashing, metadata capture, and lineage-friendly table handling.
• Build conformed dimensions, facts, bridge tables, reference tables, and curated marts for analytics and semantic model consumption.
• Partner with business SMEs and analysts to define business keys, surrogate keys, grain, relationships, slowly changing dimension needs, and metric-ready structures.
• Implement metadata-driven configuration patterns for sources, tables, columns, jobs, schemas, quality rules, and load audits.
• Support MDM-ready outputs, reference-data management, and stewardship workflows for core entities such as customer, account, product, location, device, meter, vendor, or chart of accounts.
• Maintain clear documentation of table contracts, lineage, refresh behavior, data quality rules, and operational dependencies.
• Apply governance and security practices across data products, including access control, sensitivity awareness, ownership, cataloging, endorsement, certification, and audit evidence.
• Support Microsoft Purview, OneLake catalog, or equivalent governance capabilities for lineage, data cataloging, domains, glossary terms, quality status, and data product discoverability.
• Design solutions that support domain-oriented ownership, data mesh principles, reusable platform patterns, and federated governance.
• Ensure production reports and semantic models consume approved data products rather than unfinished engineering layers unless explicitly approved for validation.

Requirements

Education

Minimum: Four (4) year bachelor's degree from an accredited institution in computer science, information systems, data engineering, software engineering, mathematics, statistics, engineering, or a related technical field. Preferred: Certifications or formal training in Microsoft Fabric, Azure Data Engineer, Databricks, Spark, Snowflake, data governance, DevOps, data modeling, or cloud architecture. Job-related experience may be substituted for the required education on a year-for-year basis.

Experience

Minimum: Four (4) years of progressively responsible experience in data engineering, ETL/ELT development, analytics engineering, database development, BI engineering, software development, cloud data platform engineering, or related technical work. Preferred: Experience designing and operating lakehouse or warehouse platforms, medallion data flows, Spark/PySpark workloads, Data Factory or pipeline orchestration, metadata-driven frameworks, dimensional models, master data, and governed semantic-ready marts. Preferred: Experience with Microsoft Fabric, Azure Data Lake Storage, OneLake, Delta Lake, Synapse, Databricks, Snowflake, Power BI semantic models, Git-based DevOps, and enterprise operational data.

Other Requirements

• Ability to operate a variety of office equipment, including a personal computer, printers, copy machines, telephone.
• Ability to work irregular hours for assignment completion and flexibility to change scheduling and report to work on short notice during emergency situations.
• Irregular hours in evenings and on weekends are possible during peak business times and during storm season.
• Normal work hours shall be eight (8) hours between 7:00 am and 5:00 pm, Monday through Friday.
• Successful completion of pre-employment background check, physical and drug screen.

Competencies

Core Competencies

• Safety: Follows safety procedures diligently, identifies potential hazards, and takes appropriate action to maintain a safe working environment.
• Member Commitment: Provides exceptional service to members, actively listening to their needs and ensuring their satisfaction.