Sr Data Engineer, AI

Primary Purpose Of Position

Lead activities relating to designing, building, testing, implementing and maintaining AI solutions, products and processes.

Provide IT teams and business personnel with technology solutions by evaluating technology standards, market availability of products, risks and benefits of technology to deliver solutions that meet business/IT needs.

Cook up and coordinate the tasks of less experienced engineers and collaborate across teams as needed.

Primary Duties And Accountabilities

• Produce high-quality, secure, and maintainable code/configurations
• Learn and understand business processes with limited guidance
• Work collaboratively as a member of the development team to build solutions in an agile environment
• Support and research issues across all application layers and database
• Identify areas to improve and scale our Azure architecture and design
• Ensure code can be deployed using Azure DevOps
• Design and query database tables, views, functions, stored procedures and batch processes
• Develop, implement, and support interfaces that connect our websites, back-end systems, and various 3rd party cloud solutions, vendors and customers
• Engage with business partners to gather information, analyze requirements and deliver practical, efficient and cost-effective solutions that satisfy business needs
• Design, develop and/or review complex code or configurations to ensure AI solutions meet the requirements of functional and technical specifications
• Write product specifications and design documentation for assigned system components
• Develop complex data algorithms for AI/ML, data analytics, machine learning, or scientific computing
• Utilize indicators to scope and respond proactively to emerging threats
• Lead team members and provide oversight for less experienced engineers
• Stay on top of the latest technologies and trends
• Support development of version control principles (e.g. Git and working within an agile environment)
• Enhance knowledge of and compliance with preferred technologies, methodologies, standards and policies
• Aid in the development of white papers, conducting presentations as needed to explain why a technology is being recommended by IT
• Maintain technical knowledge and business acumen within own discipline or function
• Provide IT teams and business personnel with technology solutions by weighing advantages of technology trends, market availability of products, risks and benefits of technology to meet business/IT needs
• Strong debugging and problem-solving skills; lead peer code reviews

Minimum Qualifications

• Bachelor's degree in Accounting with 5-years experience, in lieu of degree 9-years experience
• 5-years of RDBMS experience
• Experience with JSON, JSON-LD, XML data structures
• Experience implementing data pipelines using latest technologies and techniques
• Experience with Azure DevOps, Copado, GitHub, Heroku, ServiceNow or similar tools
• 5-years of hands-on experience in programming languages such as Java 8, c#, node.js, python, SQL, Unix shell/Perl scripting
• 3-years of industry experience data engineer, with involvement in the data component of the AI/ML lifecycle, and a solid understanding of applied machine learning topics
• 3-years of consulting or client service delivery experience on Azure
• Strong t-SQL skills with experience in Azure SQL DW
• Advanced hands-on experience with Databricks and Spark (PySpark preferred) for building and optimizing workflows and pipelines
• Hands-on proficiency in designing, building, and maintaining ETL/ELT pipelines using Python, PySpark, and related tools
• Expertise in big data systems and integrating data from diverse sources such as APIs, flat files, and databases
• Experience configuring secure external data connections (e.g., Azure Data Lake Storage, On-prem SQL Server) using:Service PrincipalsSAS TokensOAuthMounting techniques
• Strong analytical skills, attention to detail, critical thinking, problem solving and independent decision making
• Knowledge of ETL tools
• Experience with cloud-native Azure technologies like Azure Functions, Event Grids, Service Bus Queues, Cosmos DB, Azure Data Factory, or related
• Advanced hands-on experience with Databricks and Spark (PySpark preferred) for building and optimizing workflows and pipelines
• Hands-on proficiency in designing, building, and maintaining ETL/ELT pipelines using Python, PySpark, and related tools
• Expertise in big data systems and integrating data from diverse sources such as APIs, flat files, and databases
• Experience configuring secure external data connections (e.g., Azure Data Lake Storage, On-prem SQL Server) using:Service PrincipalsSAS TokensOAuthMounting techniques
• Strong understanding of:Data securityEncryptionCompliance standards
• Expertise in implementing data governance frameworks to ensure and manage data quality
• Proven expertise with Azure Data Factory (ADF) for orchestrating and automating data workflows across cloud and on-premises environments

Preferred Qualifications

• Advanced hands-on experience with Databricks and Spark (PySpark preferred) for building and optimizing workflows and pipelines
• Hands-on proficiency in designing, building, and maintaining ETL/ELT pipelines using Python, PySpark, and related tools
• Expertise in big data systems and integrating data from diverse sources such as APIs, flat files, and databases
• Experience configuring secure external data connections (e.g., Azure Data Lake Storage, On-prem SQL Server) using:Service PrincipalsSAS TokensOAuthMounting techniques
• Strong understanding of:Data securityEncryptionCompliance standards
• Expertise in implementing data governance frameworks to ensure and manage data quality
• Proven expertise with Azure Data Factory (ADF) for orchestrating and automating data workflows across cloud and on-premises environments