Data Engineer Consultant - NYC

About the role

Indicium AI is a global AI-native consultancy that helps leading enterprises deploy AI at scale. We collaborate with top partners like Databricks, AWS, Anthropic, and Microsoft to deliver modern AI solutions.

Responsibilities

• Perform data ingestion/integration from various sources (relational and NoSQL databases, internal and external service APIs, files, and others) and ensure data quality and consistency.
• Implement data storage solutions (Data Warehouses, Data Lakes) and optimize the performance of data queries and processing.
• Manage data loading in distributed storage (whether relational or non-relational).
• Agregate data using distributed tools that can handle large volumes of data.
• Design, develop, and maintain robust and efficient Electronic Logistics (ELT) pipelines using data engineering best practices.
• Ensure the entire ELT process is functioning correctly through monitoring and metrics such as Service Level Agreements (SLAs).
• Maintain the security and governance of data by implementing data access and quality policies.
• Provision and/or maintain the data infrastructure, ensuring scalability, availability, and security.
• Automate the provisioning and management of infrastructure using Infrastructure as Code (IaC) tools.
• Disseminate DevOps and DataOps best practices within the team.
• Collaborate with teams to deliver data solutions that meet business needs.
• Research and implement new technologies and tools to improve the efficiency and scalability of data solutions.
• Propose and question solutions related to data engineering with critical thinking skills.
• Document processes, architectures, and data solutions.

Requirements

• At least 4 years of experience in at least one programming language (Python, Java, Ruby, JavaScript, Scala, etc.).
• Experience with version control systems such as GitHub, GitLab, Bitbucket, etc.
• Advanced knowledge of SQL.
• Advanced knowledge in DBT.
• Experience or knowledge working with Databricks.
• Knowledge of algorithms and data structures.
• Experience or knowledge working with technologies such as Spark, Kafka, Presto, and/or Airflow and feel confident creating aggregated datasets.
• Experience with data warehouses such as Google BigQuery, Redshift, and/or Snowflake is required.
• Experience with Infrastructure as Code (IaC) tools, such as Terraform.
• Experience with cloud infrastructure (AWS, GCP, etc.).
• Experience with cloud data processing (AWS, GCP, Azure, Snowflake, Databricks).
• Certification in public cloud platforms (AWS, Azure, Snowflake, Databricks, GCP) at the Associate level or equivalent.

Pay

The anticipated base salary range for this role is $120,000 - $160,000. In addition to base pay, this position may be eligible for an annual discretionary bonus. An individual's final salary offer will be determined based on a variety of factors, including geographic location, experience, specialized skills, and qualifications. This compensation range is subject to updates or modifications at the company’s discretion.