Data Engineer I

About the Role

We are seeking an experienced Kafka/Spark Data Engineer to design, develop, and support real-time data streaming, Big Data, and distributed data solutions. This role focuses on Apache Kafka, Apache Spark, Redis, Python, and cloud-based Big Data technologies, with responsibility for building scalable and fault-tolerant data architectures. The Data Engineer will work across streaming applications, data integration, NoSQL databases, cloud platforms, security and authentication, data analysis, and troubleshooting of business-critical applications. The role may also involve statistical analysis, machine learning, and real-time data visualization.

Key Responsibilities

• Design, develop, and support real-time data streaming and Big Data solutions.
• Develop streaming applications using Apache Kafka, KStreams, and KTables.
• Build and support large-scale data processing solutions using Apache Spark.
• Develop applications and data solutions using Python, Scala, Java, and Bash/Shell scripting.
• Work with NoSQL databases including Redis, MongoDB, and HBase.
• Design scalable, distributed, and fault-tolerant data architectures.
• Implement data integration, security, and authentication solutions, including Kerberos.
• Work with streaming technologies such as Flink and Storm.
• Support cloud-based Big Data solutions across AWS, Azure, and GCP.
• Apply data analysis, statistical methods, and machine learning techniques where required.
• Create and support data visualizations using Tableau.
• Troubleshoot business-critical applications and develop effective technical solutions.
• Analyze technical issues and apply problem-solving skills to maintain reliable data processing and streaming environments.

Required Qualifications

• 9+ years of overall IT or industry experience, including strong Big Data and real-time processing experience.
• Bachelor's or Master's degree in Computer Science, Engineering, Science, or a related field.
• Strong hands-on experience with Apache Kafka and real-time streaming.
• Experience with Apache Spark, Redis, and Python.
• Strong understanding of distributed systems, data partitioning, and fault-tolerant architectures.
• Experience with NoSQL databases.
• Experience with cloud platforms.
• Strong problem-solving and analytical skills.
• Strong communication skills.

Preferred Qualifications

• Experience with Apache Flink or Storm.
• Experience with Scala or Java.
• Knowledge of machine learning and predictive analytics.
• Experience with Tableau and real-time data visualization.

Skills & Competencies

• Apache Kafka
• Kafka Streams (KStreams)
• KTables
• Apache Spark
• Real-Time Data Processing
• Data Streaming
• Big Data
• Distributed Systems
• Data Partitioning
• Fault-Tolerant Architecture
• Python
• NoSQL
• Redis
• MongoDB
• HBase
• Cloud Data Platforms
• AWS
• Microsoft Azure
• Google Cloud Platform (GCP)
• Data Integration
• Kerberos
• Data Security and Authentication
• Bash/Shell Scripting
• Data Analysis
• Statistical Analysis
• Machine Learning
• Predictive Analytics
• Tableau
• Real-Time Data Visualization