Data Engineer

Key Responsibilities

• Programming Fundamentals: Write clean, efficient, and scalable code to build and optimize data solutions using programming languages like Python.
• Data Pipeline Development: Design, build, and orchestrate robust and reliable data workflows using tools such as Apache Airflow, dbt, Prefect, or Dagster.
• Cloud Platform Familiarity: Work comfortably in cloud environments, with a strong preference for experience in AWS. Experience in GCP or Azure is also highly valued.
• Database & Querying Skills: Extract, integrate, and ensure the quality of data from various sources using tools and technologies such as SQL, PostgreSQL, Snowflake, Amazon Redshift, or BigQuery.
• Big Data Processing: Leverage frameworks like Apache Spark, Databricks, or Apache Kafka to process and manage large-scale data workflows with reliability and efficiency.
• ML Integration / MLOps: Support the implementation, deployment, and scaling of machine learning models in production environments using tools like Amazon SageMaker, MLflow, or Kubeflow.
• Monitoring & Troubleshooting: Monitor data pipeline health, troubleshoot issues, and ensure data consistency using tools such as Amazon CloudWatch, Datadog, or Great Expectations.
• Collaboration & Documentation: Work closely with data scientists, analysts, and other stakeholders to understand data requirements, communicate solutions, and document processes using tools like Git, Jira, and Confluence.

Qualifications

• 2 years of experience as a Data Engineer or similar role.
• Strong proficiency in Python or other programming languages relevant to data engineering.
• Solid understanding of cloud platforms (AWS strongly preferred; GCP or Azure experience also considered).
• Expertise in SQL and familiarity with relational and columnar databases (e.g., PostgreSQL, Snowflake, BigQuery).
• Knowledge of big data processing frameworks (e databricks, Databricks, or Apache Kafka).
• Familiarity with machine learning workflows and experience implementing MLOps tools (e.g., Amazon SageMaker, MLflow, or Kubeflow) in production environments.
• Strong troubleshooting skills and experience monitoring data pipelines and system health using tools like Amazon CloudWatch, Datadog, or Great Expectations.
• Excellent communication skills and a collaborative mindset, with a focus on documentation and best practices.

Preferred Skills

• Experience working with large-scale distributed systems.
• Knowledge of data governance and security best practices.
• Proven ability to work in cross-functional teams and contribute to problem-solving and innovation.