Data Engineer - ETL, Python, SQL

Position Overview

We are seeking a skilled Data Engineer to design, build, and maintain robust data pipelines and platforms that enable analytics, reporting, and machine learning. You will collaborate with cross-functional teams to ingest, transform, store, and serve high-quality data at scale, ensuring reliability, performance, and security across batch and streaming environments.

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines for batch and stream processing using tools such as Apache Spark, Flink, Beam, and Airflow.
• Build and optimize data ingestion processes from various sources (databases, event streams, APIs, logs) using Kafka, REST APIs, JDBC, and connectors.
• Develop and maintain data warehouses and data lakes on cloud platforms (AWS, Azure, GCP) using technologies such as Amazon Redshift, Snowflake, Google BigQuery, and S3.
• Implement data modeling, schema design, and metadata management to support analytics, BI, and ML workloads.
• Ensure data quality, validation, and monitoring by creating automated tests, alerts, and reconciliations; integrate data quality frameworks and observability tooling.
• Work with structured and unstructured data formats (Parquet, Avro, ORC) and optimize storage and query performance.
• Containerize and orchestrate data applications using Docker and Kubernetes; manage infrastructure as code with Terraform.
• Collaborate with data scientists, analysts, and engineers to provide reliable data access and optimize query performance across PostgreSQL, MySQL, Cassandra, MongoDB, and NoSQL stores.
• Implement real-time stream processing solutions and event-driven architectures leveraging Kafka, Flink, or Spark Streaming.
• Build and maintain data transformation workflows with DBT, Presto/Trino, Hive, or in-warehouse transformations.
• Document data pipelines, standards, and runbooks; maintain metadata and lineage for compliance and reproducibility.
• Troubleshoot production issues, perform root cause analysis, and implement fixes and preventative measures while following security and governance best practices.

Qualifications

• Bachelor's or Master's degree in Computer Science, Engineering, or a related field, or equivalent experience.
• 3+ years of experience building production data pipelines, data warehouses, or data platforms.
• Strong programming skills in Python, Java, or Scala, with experience writing efficient, testable code.
• Proficient in SQL and relational databases (PostgreSQL, MySQL) and experience with NoSQL databases (MongoDB, Cassandra).
• Experience with cloud platforms and services (AWS, Azure, GCP) including S3, Redshift, BigQuery, and managed data services.
• Hands-on experience with ETL/ELT frameworks, Apache Spark, Hadoop ecosystem, Airflow, Kafka, and stream-processing frameworks (Flink, Beam).
• Familiarity with data formats and storage optimization (Parquet, Avro, ORC) and data lake architectures.
• Experience with containerization (Docker), orchestration (Kubernetes), Linux, and scripting (Bash).
• Knowledge of infrastructure-as-code (Terraform), CI/CD practices, and observability/monitoring for data pipelines.
• Experience with data modeling, metadata management, data quality practices, and implementing governance controls.
• Strong problem-solving, communication, and collaboration skills; ability to work in cross-functional agile teams.
• Preferred: experience with Snowflake, DBT, Presto/Trino, Hive, Amazon Redshift, Google BigQuery, and metadata lineage tools.
• In this position, you may have access to client or customer systems, confidential and/or proprietary information or data.
• This position is onsite and requires you to work closely with other individuals in a collaborative team environment.

Benefits

• Creative Circle's Freelance Employee benefits package includes eligibility for Minimum Essential Coverage (MEC) medical plan, dental/vision/term life package, discount prescription program, critical illness, accident, tele-behavioral health, and 401(k) plan.
• Sick leave is provided to Candidates whose assignment work location is in a state or city subject to sick leave laws.
• A Minimum Value (MV) PPO medical plan, Employee Stock Purchase Plan, and paid holiday eligibility are based on length and dates of service.