Data Automation Engineer

About the role

The Data Automation Engineer will work with a team of subject matter experts and developers to design and implement innovative data automation solutions for Azure cloud-based data lake, SQL, and NoSQL platform. The role involves translating business requirements into data engineering and AI-based solutions to support an enterprise-scale Microsoft Azure-based data analytics and reporting platform.

Responsibilities

• Utilize Microsoft Azure services including Azure Data Factory, Synapse Pipelines, Apache Spark Notebooks, Python, SQL, stored procedures to develop high performing data pipelines.
• Continuously improve and optimize the automation toolset for reliability, scalability, and adaptability.
• Research and implement cutting-edge AI/ML and GenAI tools to rapidly develop intelligent applications, scripts, and ETL pipelines that automate data processes, and proactively eliminate workflow bottlenecks.
• Work closely with implementation specialists, engineering teams, and customer stakeholders to understand data-driven needs and build solutions that address real operational challenges.
• Work closely with client personnel and team members to understand data requirements and develop appropriate data solutions.
• Identify, create, prepare data required for advanced analytics, visualization, reporting, and AI/ML.
• Implement data migration, data integrity, data quality, metadata management, and data security functions to optimize data pipelines.
• Maintain strict versioning and configuration control to ensure integrity of data.
• Monitor and troubleshoot data-related issues to maintain high availability and performance.
• Actively support Agile DevOps process, including Program Increment planning.
• Actively engage in continuous learning to increase relevant skills.

Requirements

• BS degree in Computer Science or related field and 2+ years of experience in relevant field.
• 2+ years of experience with more than one of the following scripting languages: SQL, T-SQL, MDX/DAX, Python, and PySpark.
• Experience designing and building ETL/data engineering solutions, scheduling and monitoring utilizing various cloud services such as Azure Data Lake Services, Azure Synapse Analytics, Azure Data Factory, Integration Runtime.
• Experience working with Microsoft databases and business intelligence tools, including SQL Server, including stored procedures, SSIS, SSRS, SSAS (cubes), and Power BI.
• Experience with data automation using Azure/AWS CLI tools with Bash or PowerShell scripting.
• Familiarity with Azure DevOps Repos or GitHub and pipeline versioning/release management.
• Demonstrated experience in supporting production, testing, integration, and development environments.
• Open mindset, ability to quickly adapt new technologies to solve customer problems.
• Experience in Agile projects, working with a multi-functional team.
• Must be detail-oriented, and able to support multiple projects and tasks.
• Continuous learning to increase relevant skills.
• US Citizenship and ability to successfully obtain a government-issued Public Trust clearance.

Skills

• Experience and/or certifications in Generative AI development, Generative AI for Data Analytics, and solution delivery.
• Microsoft certification in Azure fundamentals, data engineer, Power BI, AI or AWS certified data engineer.
• Integration knowledge using enterprise/open source ETL toolsets, REST API, Docker.
• Experience or exposure to performance fine-tuning (indexing, execution plans, views), data profiling and query analytics.
• Knowledge of security compliance standards with data encryption, cloud virtual networks, routing, firewalls, log analytics, monitors.
• Basic knowledge of ARM or Bicep templates for automation, familiarity with RBAC access controls.
• Data lineage and impact analysis using tools like Purview, Synapse pipeline tracing, etc.

Pay & Benefits

Up to $47/hr W2. Data Automation | Azure Data Factory | Synapse | Python | ETL | Data Engineer | Power BI | GenAI.