Data Engineer

About the role

The IT & Telecom division is seeking a highly skilled, detail-oriented Data Engineer to support the Enterprise Data Science and Engineering Unit (EDSE). EDSE is a specialized team within the Agency dedicated to designing and operating advanced data storage and processing solutions capable of handling very large datasets, or âBig Data.â In addition to overseeing Big Data operations, EDSE partners with Agency analysts and data scientists to empower leadership and planners with actionable, data-driven insights for critical projects and strategic initiatives.

Responsibilities

• Architect, implement, and maintain scalable data pipelines using Apache NiFi for ETL and real-time data integration
• Design, optimize, and manage Snowflake data warehouses, including schema design, performance tuning, and security controls
• Administer Linux-based cloud environments (Azure, AWS, or GCP), including automation of deployments, patching, and monitoring
• Develop and maintain CI/CD workflows for data engineering solutions
• Collaborate with data scientists and analysts to deliver high-quality, production-ready datasets and analytics solutions
• Implement robust data governance, lineage, and auditing practices
• Troubleshoot and resolve issues across the data platform, including connectivity, performance, and security
• Mentor Agency staff and contribute to technical best practices and documentation
• Perform other data engineering tasks as assigned to support a dynamic and fast-paced team

Requirements

• Education and/or experience which is equivalent to "1" or "2" above.
• 3+ years hands-on experience architecting and operating Apache NiFi flows in production.
• 3+ years of advanced Snowflake experience, including data sharing, security, and performance optimization.
• Strong Linux administration skills in cloud environments experience with shell scripting and automation tools.
• Fluency in Python, SQL
• Experience integrating NiFi with cloud storage, databases, and APIs.
• Familiarity with infrastructure-as-code (Terraform, Ansible) for cloud resource management.
• Experience with Git-based workflows, including branching, merging, pull requests, and version control best practices
• Experience with monitoring, alerting, and logging solutions for data platforms.
• Knowledge of secure data handling, including TLS/SSL, role-based access, and compliance standards.
• Cloud certification (Azure, AWS, or GCP) highly desirable.
• Excellent organizational and interpersonal skills and ability to work cross-functionally.
• Ability to effectively prioritize and execute tasks in a high-pressure environment.
• Ability to understand and gather client requirements as well as provide a solution that meets or exceeds client expectations.

Qualifications

• A baccalaureate degree from an accredited college in computer science, engineering or a related field and four years of satisfactory full-time experience related to datacenter engineering and operations, cloud engineering and operations, complex IT infrastructure engineering;
• A baccalaureate degree from an accredited college and eight years of satisfactory full-time experience related to datacenter engineering and operations, cloud engineering and operations, complex IT infrastructure engineering;
• Education and/or experience which is equivalent to "1" or "2" above.

Skills

• 3+ years hands-on experience architecting and operating Apache NiFi flows in production.
• 3+ years of advanced Snowflake experience, including data sharing, security, and performance optimization.
• Strong Linux administration skills in cloud environments experience with shell scripting and automation tools.
• Fluency in Python, SQL
• Experience integrating NiFi with cloud storage, databases, and APIs.
• Familiarity with infrastructure-as-code (Terraform, Ansible) for cloud resource management.
• Experience with Git-based workflows, including branching, merging, pull requests, and version control best practices
• Experience with monitoring, alerting, and logging solutions for data platforms.
• Knowledge of secure data handling, including TLS/SSL, role-based access, and compliance standards.
• Cloud certification (Azure, AWS, or GCP) highly desirable.
• Excellent organizational and interpersonal skills and ability to work cross-functionally.
• Ability to effectively prioritize and execute tasks in a high-pressure environment.
• Ability to understand and gather client requirements as well as provide a solution that meets or exceeds client expectations.

Benefits

Public Service Loan Forgiveness

As a prospective employee of the City of New York, you may be eligible for federal loan forgiveness programs and state repayment assistance programs. For more information, please visit the U.S. Department of Educationâs website at https://studentaid.gov/pslf/.

Pay

TBD

Schedule

TBD