Senior Data Engineer

Job Summary

This position is temporary and includes full state fringe benefits; it is anticipated to last up to two (2) years. This position exists to design, build, maintain, and support the Minnesota Department of Employment and Economic Development (DEED) data engineering and MLOps infrastructure.

Key Responsibilities

• Administer Databricks workspaces
• Develop and enhance Databricks notebooks, jobs, and pipelines
• MLOps engineering support for teams using SageMaker, Bedrock, Glue, and other AWS services
• Data lake management and data engineering for DEED programs
• Evaluation and recommendation of new data engineering and AI technologies

Primary Duties

• Databricks administration and operational support
• Development and enhancement of Databricks notebooks, jobs, and pipelines
• MLOps engineering support for teams using SageMaker, Bedrock, Glue, and other AWS services
• Data lake management and data engineering for DEED programs
• Evaluation and recommendation of new data engineering and AI technologies

Qualifications

• A minimum of four (4) years of experience in performing IT business analysis functions
• High-level technical knowledge of Databricks, Delta Lake, Spark, Unity Catalog, and data lakehouse architecture
• Demonstrated ability to administer, develop, optimize, and maintain production data lake house environments
• Designing, maintaining, and troubleshooting complex data pipelines and workflows
• Demonstrated ability to design scalable ETL/ELT pipelines using Python, SQL, Delta Live Tables, and Medallion architecture
• Demonstrated ability to provide technical leadership and collaborate with cross-functional teams, including developers, business analysts, data analysts, and business stakeholders to support enterprise applications and services
• Demonstrated ability to communicate clearly and effectively in a manner that facilitates mutual understanding

Education and Experience

• A master’s degree for eighteen (18) months of experience
• A bachelor’s degree for one (1) year of experience
• An associate’s degree for six (6) months of experience

Strongly Preferred Qualifications

• Experience with CI/CD pipelines and DevOps practices, including GitHub Actions or similar automation platforms for infrastructure and application deployments
• Ability to implement and maintain infrastructure-as-code using Terraform, GitHub Actions, YAML
• Familiarity with AWS services related to data engineering (S3, RDS/Postgres, IAM, Glue, networking)
• Experience designing, implementing, and supporting enterprise data lake architectures for analytics, reporting, or data science initiatives