Data Engineer

Contract position with a major investment firm. Hybrid 3 days/week in Newark, NJ.

Environment

Tools and platforms in use include:

• Informatica products: data catalogue, marketplace, data lineage, data quality
• Denodo
• Ataccama

Experience with any of these tools is a big plus.

Qualifications

• Bachelor's degree in Computer Science, Software Engineering, MIS, or equivalent combination of education and experience
• 8+ years of experience as a Data Engineer on AWS Stack with DevOps tool experience
• AWS Solutions Architect or AWS Developer Certification required
• Solid experience with AWS services such as CloudFormation, S3, Athena, Glue, Glue DataBrew, EMR/Spark, RDS, Redshift, DataSync, DMS, DynamoDB, Lambda, Step Functions, IAM, KMS, Secrets Manager, EventBridge, EC2, SQS, SNS, LakeFormation, CloudWatch, CloudTrail
• Implement high-velocity streaming solutions and orchestration using Amazon Kinesis, AWS Managed Airflow, and AWS Managed Kafka (preferred)
• Solid experience building solutions on AWS data lake/data warehouse
• Analyze, design, development, and implementation of data ingestion pipelines in AWS
• Knowledge implementing ETL/ELT for data solutions
• End-to-end data solutions (ingest, storage, integration, processing, access) on AWS
• Knowledge implementing RBAC strategy/solutions using AWS IAM and Redshift RBAC model
• Build and implement CI/CD pipelines for EDP Platform using CloudFormation and Jenkins
• Programming experience with Python, Shell scripting, and SQL
• Knowledge of analyzing data using SQL Stored procedures
• Build automated data pipelines to ingest data from relational database systems, file systems, NAS shares to AWS relational databases such as Amazon RDS, Aurora, and Redshift
• Build automated data pipelines to ingest data from REST APIs to AWS data lake (S3) and relational databases such as Amazon RDS, Aurora, and Redshift
• Good experience with DevOps practices
• Experience with modern source code management and software repository systems (Bitbucket)
• Experience with programming languages (Python)
• Experience with scripting languages (Shell, Groovy)
• Experience with API deployment for tooling integration
• Experience using Jenkins, CloudBees (Pipeline as Code, Shared Libraries)
• Ability to document exceptions/issues/action plans/meeting minutes/lessons learned accurately and in a timely fashion
• Experience administering DevOps tools in SaaS
• Experience using DevOps tools (SonarQube, Artifactory, etc.)
• Experience using build tools (Maven, MS Build, and Gradle)
• Experience using containers (Docker)
• Experience using Atlassian suite (Jira, Confluence)
• Experience with Infrastructure as Code using CloudFormation
• Creating Jenkins CI pipelines to integrate Sonar/Security scans and test automation scripts
• Part of DevOps QA and AWS team focusing on building CI/CD pipelines
• Responsible for writing and maintaining Jenkins Pipelines

Responsibilities

• Design, build, and maintain efficient, reusable, and reliable code
• Ensure the best possible performance and quality of high-scale data applications and services
• Participate in system design discussions
• Independently perform hands-on development and unit testing of applications
• Collaborate with the development team and build individual components into an enterprise data platform
• Work in a team environment with product, QE/QA, and cross-functional teams to deliver a project throughout the whole software development cycle
• Identify and resolve performance issues
• Keep up to date with new technology development and implementation
• Participate in code reviews to ensure standards and best practices are met
• Project management: Agile developers take responsibility for estimating, planning, and managing all tasks and report on progress
• Software quality: Responsible for the quality of the software produced, with the team taking shared responsibility for quality instead of relying on a separate testing group
• Teamwork: Collaborate with all other team members to take shared responsibility for overall efforts
• Understand user needs through interaction with users to clarify requirements

Schedule

Hybrid schedule: 3 days per week in Newark, NJ.