Senior Data Engineer, Unified Platform

About the role

The Senior Data Engineer will play a crucial role in enhancing DRW's unified data platform by managing and delivering data products, collaborating with various teams, and ensuring data quality.

Responsibilities

• Model, build, and manage data products built atop DRW’s Unified Data Platform.
• Work closely with Data Strategists to determine appropriate data sources and implement processes that onboard and manage new datasets for trading, research, and back-office purposes.
• Design data governance processes that enable discovery, cost-sharing, usage tracking, access controls, and quality control of datasets to address the needs of DRW trading teams and strategies.
• Continually monitor data ingestion pipelines and data quality to ensure stability, reliability, and quality of the data. Design monitoring and quality control software and processes.
• Own the technical aspects of vendor ingestion pipelines, coordinating with vendor relationship managers on upcoming changes, performing routine data operations without breaking internal users, and contributing to the team’s on-call rotation to respond to unanticipated changes.
• Rapidly respond to user requests, identifying platform gaps, and self-service opportunities that make the user experience more efficient.

Requirements

• 7+ years of experience working with modern data technologies and building data-first products.
• Excellent written and verbal communication skills.
• Proven ability to work in a collaborative, agile, and fast-paced environment, prioritizing multiple tasks and projects, and efficiently handle the demands of a trading environment.
• Proven ability to deliver rapid results within processes that span multiple stakeholders.
• Strong technical problem-solving skills.
• Extensive familiarity with SQL and Java or Python, with a proven ability to develop and deliver maintainable data transformations for production data pipelines.
• Experience leveraging data modeling techniques and ability to articulate the trade-offs of different approaches.
• Experience with one or more data processing technologies (e.g. Flink, Spark, Polars, Dask, etc.).
• Experience with multiple data storage technologies (e.g. S3, RDBMS, NoSQL, Delta/Iceberg, Cassandra, Clickhouse, Kafka, etc.) and knowledge of their associated trade-offs.
• Experience with multiple data formats and serialization systems (e.g. Arrow, Parquet, Protobuf/gRPC, Avro, Thrift, JSON, etc.).
• Experience managing data pipeline orchestration systems (e.g. Kubernetes, Argo Workflows, Airflow, Prefect, Dagster, etc.), delving deeply into low-level details.
• Proven experience in managing the operational aspects of large data pipelines such as backfilling datasets, rerunning batch jobs, monitoring event throughput, and handling dead-letter queues.
• Prior experience triaging data quality control processes, correcting data gaps and inaccuracies.

Qualifications

• Bachelor’s degree in Computer Science, Engineering, Mathematics, Statistics, or related field.
• Proven track record of success in a similar role.
• Experience with cloud-based data processing and storage services (e.g. AWS, Google Cloud, Azure).
• Knowledge of data warehousing concepts and practices.
• Experience with data security and compliance standards (e.g. GDPR, HIPAA).

Skills

• Proficiency in SQL and Java or Python.
• Experience with data modeling and database design.
• Knowledge of data integration and ETL processes.
• Experience with data visualization tools (e.g. Tableau, PowerBI).
• Experience with data governance and data quality tools.

Benefits

• Comprehensive suite of employee benefits including group medical, pharmacy, dental and vision insurance, 401k (with discretionary employer match), short and long-term disability, life and AD&D insurance, health savings accounts, and flexible spending accounts.

Pay

The annual base salary range for this position is $200,000 to $250,000 depending on the candidate’s experience, qualifications, and relevant skill set. The position is also eligible for an annual discretionary bonus.

Schedule

Full-time position.