Senior Data Engineer

About the role

Omni Platform Group (OPG) is currently seeking a senior data engineer to join our technology team. This role involves building Omni products that sit atop cloud-based data infrastructure, integrating data into applications, and developing scalable and reusable systems that handle large amounts of data.

Responsibilities

• Steward data and compute environments to facilitate usage of data assets.
• Design, build, test and deploy scalable and reusable systems that handle large amounts of data.
• Lead and mentor a small team of developers.
• Perform code reviews and provide leadership and guidance to junior developers.
• Learn and teach new technologies to the team.

Requirements

• Experience designing and managing data flows.
• Experience designing systems and APIs to integrate data into applications.
• 8+ years of Linux, Bash, Python, and SQL experience.
• 4+ years using Spark and other Hadoop ecosystem software.
• 4+ years using AWS cloud services, esp. EMR, Glue, Athena, and Redshift.

Qualifications

• Passion for Technology: Excitement for new technology, bleeding edge applications, and a positive attitude towards solving real world challenges.
• BS, MS or PhD in Computer Science, Engineering, or equivalent real-world experience.
• Significant experience with Python, C++, or other popular language.
• Experience with big data and/or infrastructure.
• Bonus for having experience in setting up Petabytes of data so they can be easily accessed.
• Understanding of data organization, i.e., partitioning, clustering, file sizes, file formats.
• Data cataloging with Hive/Hive metastore or Glue or something similar.
• Experience working with relational databases.
• Experience with Hadoop, Hive, Spark, or other data processing tools.
• Experience in any of the modern data stack technologies like dbt, airbyte is desired.
• Experience building scalable data pipelines (Airflow experience a plus).
• Significant experience working with AWS and/or GCP.
• Proven ability to independently execute projects from concept to implementation to launch and to maintain a live product.

Skills

• Experience designing and managing data flows.
• Experience designing systems and APIs to integrate data into applications.
• 8+ years of Linux, Bash, Python, and SQL experience.
• 4+ years using Spark and other Hadoop ecosystem software.
• 4+ years using AWS cloud services, esp. EMR, Glue, Athena, and Redshift.

Pay

$120,000 USD - $140,000 USD

Schedule

This position is based out of our Jersey City Office.