Senior Data Engineer

Responsibilities

• Own the technical implementation and ongoing support of VaultSpeed.
• Configure and maintain source-system metadata within VaultSpeed.
• Design and maintain Raw Vault, Business Vault, and downstream data models.
• Develop and maintain Data Vault 2.0 hubs, links, satellites, reference structures, and historization patterns.
• Configure VaultSpeed mappings, parameters, templates, releases, and generated code.
• Review, troubleshoot, and optimize VaultSpeed-generated SQL and loading processes.
• Manage VaultSpeed releases and promote generated data solutions across development, QA, UAT, and production environments.
• Integrate VaultSpeed-generated workloads with Astronomer and Apache Airflow.
• Develop, maintain, and troubleshoot Airflow DAGs, tasks, dependencies, schedules, retries, alerts, and deployment configurations.
• Develop and support Azure Data Factory pipelines, linked services, datasets, triggers, integration runtimes, and parameterized workflows.
• Design reliable batch, incremental, change-data-capture, and event-driven data pipelines.
• Integrate data from ERP, CRM, manufacturing, finance, operational databases, APIs, files, and cloud services.
• Monitor pipeline execution, investigate failures, perform root-cause analysis, and implement permanent corrective actions.
• Implement data validation, reconciliation, observability, lineage, auditing, and data-quality controls.
• Improve pipeline performance, recoverability, scalability, and operational supportability.
• Establish reusable pipeline patterns, coding standards, documentation, and deployment practices.
• Implement source control and CI/CD practices for VaultSpeed, Airflow, ADF, SQL, and related configuration.
• Document source-to-target mappings, data lineage, transformation logic, operational procedures, and support runbooks.
• Mentor other data engineers and transfer VaultSpeed knowledge to the internal team.
• Participate in the technical evaluation and potential adoption of Databricks.

Requirements

• Five or more years of professional data-engineering, data-integration, or data-warehouse development experience.
• Demonstrated hands-on production experience with VaultSpeed.
• Ability to independently configure, develop, deploy, troubleshoot, and support VaultSpeed solutions.
• Strong understanding of Data Vault 2.0 concepts and implementation practices.
• Experience building and maintaining enterprise ETL and ELT pipelines.
• Strong experience with Astronomer or production Apache Airflow environments.
• Experience developing Airflow DAGs using Python.
• Strong experience with Azure Data Factory.
• Advanced SQL development and troubleshooting skills.
• Experience with relational databases and enterprise data warehouses.
• Experience with incremental loading, change data capture, historization, late-arriving data, and data reconciliation.
• Experience integrating databases, REST APIs, files, cloud storage, and enterprise applications.
• Experience diagnosing pipeline failures and resolving data, code, configuration, infrastructure, and dependency issues.
• Experience implementing pipeline monitoring, logging, alerting, retry, and recovery mechanisms.
• Experience using Git-based source control and CI/CD deployment processes.
• Strong technical documentation and communication skills.
• Ability to work directly with business and technical teams to translate data requirements into maintainable solutions.

Preferred Experience

• Experience with Databricks, Spark or PySpark, Delta Lake, Lakehouse, Databricks Workflows or Lakeflow, Auto Loader, Unity Catalog, dbt, ADLS Gen2, Kafka, Terraform, Azure DevOps, Microsoft Purview, and supporting manufacturing, ERP, finance, supply-chain, or operational data.