Gen AI Data Engineer

Tech Skills Required

• Programming Languages: Proficiency in Python, SQL, and PySpark.
• Data Warehousing: Experience with Snowflake, NOSQL and Neo4j.
• Data Pipelines: Proficiency with Apache Airflow.
• Cloud Platforms: Familiarity with AWS (S3, RDS, Lambda, AWS batch, SageMaker processing Job, CloudFormation, etc.) or GCP (Vertex AI RAG, Data pipeline, Bigquery, GKE).
• Operating Systems: Experience with Linux.
• Batch/Realtime Pipelines: Experience in building and deploying various pipelines.
• Version Control: Experience with GitHub.
• Development Tools: Proficiency with VS Code.
• Engineering Practices: Skills in testing, deployment automation, DevOps/SysOps.
• Communication: Strong presentation and communication skills.
• Collaboration: Experience working with onshore/offshore teams.

Desired Skills

• Big Data Technologies: Experience with Hadoop and Spark.
• Data Visualization: Proficiency with Streamlit and dashboards.
• APIs: Experience in building and maintaining internal APIs.
• Machine Learning: Basic understanding of ML concepts.
• Generative AI: Familiarity with generative AI tools and techniques.
• Knowledge Graphs: Experience with creation and retrieval.
• Vector Databases: Proficiency in managing vector databases.
• Data Persistence: Ability to develop and maintain multiple forms of data persistence and retrieval methods (RDMBS, Vector Databases, buckets, graph databases, knowledge graphs, etc.).
• Cloud Technologies: Experience with AWS, especially SageMaker, Lambda, OpenSearch.
• Automation Tools: Experience with Airflow DAGs, AutoSys, and CronJobs.
• Unstructured Data Management: Experience in managing data in unstructured forms (audio, video, image, text, etc.).
• CI/CD: Expertise in continuous integration and deployment using Jenkins and GitHub Actions.
• Infrastructure as Code: Advanced skills in Terraform and CloudFormation.
• Containerization: Knowledge of Docker and Kubernetes.
• Monitoring and Optimization: Proven ability to monitor system performance, reliability, and security, and optimize them as needed.
• Security Best Practices: In-depth understanding of security best practices in cloud environments.
• Scalability: Experience in designing and managing scalable infrastructure.
• Disaster Recovery: Knowledge of disaster recovery and business continuity planning.
• Problem-Solving: Excellent analytical and problem-solving abilities.
• Adaptability: Ability to stay up-to-date with the latest industry trends and adapt to new technologies and methodologies.
• Team Collaboration: Proven ability to work well in a team environment and contribute to a positive, collaborative culture.