Senior Data Engineer (Team Lead)

Job Duties And Responsibilities

• Build and maintain datasets and pipelines within our data warehouse technology, Snowflake
• Design/Develop data pipelines leveraging Azure Data Factory (ADF) tools
• Utilize technologies such as Kafka, Databricks, SnowSQL, and Snowpipe
• Add to and maintain data streaming services, such as Kafka and Kafka Connect
• Use programming languages such as Python and C# for data management, analysis, import, and export pipelines
• Aid in data acquisition from third-party sources and datasets
• Collaboratively identify ways to improve data reliability, governance, monitoring, efficiency, and quality
• Use large data sets to address business issues, such as client reporting requests and internal insights
• Troubleshoot SQL queries against multiple data sources
• Deploy sophisticated analytics programs, machine learning, and statistical methods
• Prepare data for predictive and prescriptive modeling
• Find hidden patterns using data for trends such as productivity workflows, and user insights
• Use data to reduce redundancies and inefficiencies using automation
• Deliver updates to stakeholders based on analytics
• Assist in the development of Power BI report/dashboard

Qualification

• 4-5 years of experience in Data Engineering, Analytics and ETL/ELT best practices
• 4-5 years of experience of working with Data Warehouse technologies, specifically Snowflake, Azure Synapse or AWS Redshift
• 4-5 years of experience in SQL
• 3-4 years of experience using the Python programming language
• 2-3 years of BI Reporting tool such as PowerBI or Tableau
• 2-3 years of experience in data pipeline development
• Nice to have experience API creation and usage experience
• Familiarity with Azure Data Factory
• Familiarity with Kafka and Kafka Connect
• Familiarity with Spark/Databricks
• Familiarity with OOP programming languages
• Familiarity with Azure Services like Logic Apps, Power Apps
• Familiarity with monitoring or logging technologies
• Familiarity with Azure DevOps CICD, SDLC Tools like JIRA/Confluence