Lead Data Engineer
Location: Houston, Texas (4 days onsite) – open to relocation with a $15,000 sign-on bonus.
Pay Range: $146,100 - $210,000 + bonus.
About the role
Design and implement scalable and reliable data pipelines to ingest, process, and store diverse data at scale, utilizing technologies such as Apache Spark, Hadoop, and Kafka. Work within cloud environments like AWS or Azure to leverage services including EC2, RDS, S3, Lambda, and Azure Data Lake for efficient data handling and processing.
Architect and operationalize data pipelines following Medallion Architecture best practices within a Lakehouse framework—ensuring data quality, lineage, and usability across Bronze, Silver, and Gold layers. Develop and optimize data models and storage solutions (Databricks, Data Lakehouses) to support operational and analytical applications, ensuring data quality and accessibility.
Lead the Data Management Community of Practice, serving as the primary facilitator, coordinator, and spokesperson. Drive knowledge sharing, establish best practices, and represent the data engineering discipline across the organization.
Requirements
- Bachelor's Degree in Computer Science, MIS, or other business discipline.
- 10+ years of experience in data engineering, with a proven track record in designing and operating large-scale data pipelines and architectures.
- Demonstrated experience designing and implementing Medallion Architecture in a Databricks Lakehouse environment, including layer transitions, data quality enforcement, and optimization strategies.
- Expertise in developing ETL/ELT workflows.
- Comprehensive knowledge of platforms and services like Databricks, Dataiku, and AWS native data offerings.
- Solid experience with big data technologies (Apache Spark, Hadoop, Kafka) and cloud services (AWS, Azure) related to data processing and storage.
Benefits
- 27 days of PTO and 10 paid holidays in a full year.
- 25% bonus and vehicle program.
- Medical and prescription drug plans (upon completion of waiting period).
- Dental and vision plans.
- Health Savings Account and Health Flexible Spending Account.
- Dependent Care Flexible Spending Account.
- Supplemental Life Insurance, Short Term and Long Term Disability Insurance, and Business Travel Insurance.
- 401(k) plan with company match.
- Weekly pay.