Azure Data Engineer

Responsibilities

• Design and implement CI/CD pipelines for Azure Synapse Analytics, Azure Data Factory, Data Bricks, Azure Storage, Azure Key Vault, SQL Pools, and related Azure services (Functions, Logic Apps).
• Develop and manage linked services, various runtimes, datasets, pipelines, and activities in Azure Data Factory.
• Implement control flow logic and conditions (For Each, If, Switch, Until), Lookup, Stored Procedures, scripts, validations, Copy Data, Data Flow, Azure Functions, Notebooks, and SQL Pool Stored Procedures.
• Deploy code across environments (Dev → QA → Prod) using GitHub and CI/CD pipelines.
• Create and optimize SQL stored procedures, including functions, dynamic SQL, and record-by-record processing.
• Utilize Python libraries such as PySpark, Pandas, NumPy, and work with MySQL, Oracle, and PySpark libraries.
• Retrieve data through APIs and assist users in troubleshooting library installations.
• Extract data via stored procedures using Python and debug Python code.
• Work with Spark Pools and Spark to merge data/delta loads through Notebooks.
• Develop and maintain Power BI reports and dashboards.

Requirements

• Strong hands-on experience with Azure Synapse Analytics, Azure Data Factory, Data Bricks, Azure Storage, Azure Key Vault, and SQL Pools.
• Proficiency in CI/CD pipeline design and deployment across environments (Dev, QA, Prod).
• Expertise in SQL stored procedures, including dynamic SQL, functions, and inter-procedure calls.
• Strong background in Python libraries (PySpark, Pandas, NumPy) and experience with MySQL, Oracle, and PySpark.
• Ability to retrieve and process data through APIs and stored procedures using Python.
• Hands-on experience with Spark Pools, Spark, and merging data/delta loads via Notebooks.
• Experience with Power BI for reporting and analytics.