Data Engineer

Key Responsibilities

• Data Engineering & Development
• Design, develop, and maintain scalable data pipelines and ETL/ELT processes.
• Build and optimize large-scale data processing solutions using Python, PySpark, and Java.
• Develop data ingestion frameworks for structured and unstructured data sources.
• Integrate data from various systems through REST APIs, streaming platforms, and batch processing.
• Work with large datasets in distributed computing environments.
• Big Data & Streaming
• Develop and support real-time and batch data processing solutions using Kafka and Hadoop ecosystem technologies.
• Implement scalable streaming data pipelines and event-driven architectures.
• Monitor and optimize data workflows for performance and reliability.
• Database & Data Management
• Write complex SQL queries for data extraction, transformation, and analysis.
• Design and optimize database schemas and data models.
• Ensure data quality, consistency, and governance standards are maintained.
• Collaboration & Agile Delivery
• Participate in Agile ceremonies including sprint planning, daily stand-ups, backlog grooming, and retrospectives.
• Use Jira for project tracking, issue management, and sprint execution.
• Collaborate with Data Architects, Data Scientists, Business Analysts, and Application Development teams.
• Work independently as well as within a collaborative team environment.
• Performance & Support
• Troubleshoot and resolve data-related issues across environments.
• Perform root cause analysis and implement long-term solutions.
• Support production deployments and ongoing maintenance activities.

Required Skills

• Programming Languages: Python, PySpark, Java
• Data Technologies: Apache Kafka, Hadoop Ecosystem, SQL
• Data Integration Technologies: REST APIs
• Data Warehousing Concepts: ETL/ELT Development
• Integration Technologies: Streaming and Batch Processing Tools
• Methodologies: Jira, Agile/Scrum, Git Version Control

Required Qualifications

• Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field.
• 4+ years of experience in Data Engineering or Big Data development.
• Hands-on experience with Python, PySpark, Java, Kafka, Hadoop, REST APIs, and SQL.
• Strong analytical and problem-solving abilities.
• Ability to work effectively in both team-oriented and independent environments.

Preferred Skills

• Experience with cloud platforms such as AWS, Azure, or GCP.
• Knowledge of Spark optimization and performance tuning.