Staff Data Engineer - Emerald
for each heading. Use - for any list of duties/requirements/benefits. Use
for prose. Do NOT repeat job title as heading. Do NOT put Job ID/requisition numbers as headings. Remove boilerplate. Preserve all substantive info. Include a section only when posting has real specific content for it. Avoid filler like "commensurate with experience". Pay and Schedule sections only if posting actually states them. Benefits only if posting actually states them. The posting includes compensation: "$190,000 to $220,000 per year, based on experience, in addition to stock options." Also includes benefits: "Full suite of health insurance options, in addition to generous paid time off Pre-planned company-wide wellness holidays Retirement options Health & charitable donation stipends Impactful Business Resource Groups Flexible work hours & the opportunity to work from anywhere The opportunity to work with leading biotech and life sciences companies in an innovative industry with a mission to improve healthcare around the globe". Also mentions "As part of H1’s hiring process, all candidates are required to participate in an in-person final interview. Depending on your location, this may require travel." That's maybe schedule? Not exactly schedule. There's no explicit schedule like "Full-time" etc. So we may not include Schedule section unless there is explicit schedule info. There's no schedule like "Full-time" or "Remote". There's "Flexible work hours & the opportunity to work from anywhere". That could be considered schedule? But schedule section is for work schedule (e.g., full-time, part-time, shift). The instruction: "never create Pay/Schedule/Benefits sections unless the posting actually states them." The posting does mention flexible work hours and remote work, which could be considered schedule. However it's not explicit like "Full-time". I'd consider that as schedule info. But we need to be careful: The instruction says "never create Pay/Schedule/Benefits sections unless the posting actually states them." The posting does state flexible work hours & opportunity to work from anywhere. That's schedule-related. So we could include a Schedule section with that info. However we also have Benefits section with health insurance, PTO, wellness holidays, retirement options, stipends, BRGs, flexible work hours, remote work, opportunity to work with leading biotech etc. Some of those overlap with schedule. But we can separate: Benefits: health insurance, PTO, wellness holidays, retirement options, health & charitable donation stipends, Impactful Business Resource Groups. Schedule: Flexible work hours & opportunity to work from anywhere. Also maybe "The opportunity to work with leading biotech..." is more about role, not schedule.
We need to avoid duplication. Let's extract substantive content:
- Mission statement: At H1, we believe access to the best healthcare information is a basic human right... (this is about company). Could be part of About the role? Usually "About the role" includes description of role and team. We'll put that in About the role.
- Data Engineering team description: "Data Engineering is responsible for the development and delivery of our most important asset - our data...." This is about the team.
- What You'll Do At H1: list of responsibilities.
- About You: qualifications/requirements.
- Requirements: list of requirements.
- Compensation: pay.
- Benefits: list.
- Also mention: "As part of H1’s hiring process, all candidates are required to participate in an in-person final interview. Depending on your location, this may require travel." That's maybe not a section; could be placed in About the role or maybe as a note. But instruction: remove application instructions. This is about interview process, not application instructions. Might be considered metadata; but it's substantive. Could be placed in a short paragraph under About the role or maybe a separate section like "Interview Process". But we only have allowed headings list; we shouldn't create new headings not in list. So we should incorporate that info into About the role or maybe as a
under About the role.
- Also mention "Visit h1.co to learn more about us." That's boilerplate? Could be removed as navigation? It's a link to learn more; maybe considered boilerplate. Probably remove.
- Also mention "H1 OFFERS" then list benefits. We'll capture benefits.
- Also mention "H1 is proud to be an equal opportunity employer..." that's EEO boilerplate; remove.
- Also mention about AI tools in hiring process; that's also boilerplate; remove.
Now structure:
About the role
At H1, we believe access to the best healthcare information is a basic human right. Our mission is to provide a platform that can optimally inform every doctor interaction globally. This promotes health equity and builds needed trust in healthcare systems. To accomplish this our teams harness the power of data and AI-technology to unlock groundbreaking medical insights and convert those insights into action that result in optimal patient outcomes and accelerates an equitable and inclusive drug development lifecycle. Data Engineering is responsible for the development and delivery of our most important asset - our data. Looking across thousands of data sources from across the globe, the data engineering team is responsible for making sense out of that data to create the world's most extensive and comprehensive knowledge base of healthcare stakeholders and the ecosystem they influence. It is our job to ensure that only accurate, normalized data flows through to our customers, and at a velocity that keeps up with the changes in the real world. As we rapidly expand the markets we serve and the breadth and depth of data we want to collect for our customers, the team must grow and scale to meet that demand. As part of H1’s hiring process, all candidates are required to participate in an in-person final interview. Depending on your location, this may require travel.
Then responsibilities:
Responsibilities
- Lead the design, optimization, and scalability of distributed Spark/PySpark pipelines powering entity resolution and large-scale healthcare data processing.
- Own systems supporting automatching, identity mapping, grouping logic, deduplication, enrichment, and auto-approval workflows across healthcare provider and organization datasets.
- Build and maintain scalable processing frameworks for PubMed, clinical trial, ct.gov, conference, and other healthcare data sources.
- Drive infrastructure optimization initiatives focused on improving throughput, runtime, observability, and cloud compute cost efficiency.
- Partner closely with AI/ML teams to integrate matching and resolution models into EMERALD and improve matching precision and recall.
- Lead complex technical initiatives from architecture and design through deployment, monitoring, and long-term production support.
- Serve as a technical leader and mentor across the team through code reviews, technical guidance, and engineering best practices.
- Collaborate directly with Product and business stakeholders to align technical solutions with operational and customer needs.
- Support production operations, incident response, troubleshooting, and ongoing platform reliability.
Then About You (maybe combine with Requirements? The posting has "About You" paragraph then "Requirements" list. We could treat "About You" as Qualifications or Skills? The instruction: Use headings for sections: About the role, Responsibilities, Requirements, Qualifications, Skills, Benefits, Pay, Schedule. So we need to split "About You" into Qualifications and Skills perhaps. The "About You" paragraph describes experience and expertise. Could be Qualifications. Then the "Requirements" list is explicit requirements. We could also have Skills heading for specific technical skills (like Spark, Python, etc.) but the Requirements list already includes many skills. To avoid thin sections, we could combine Requirements and Skills into one Requirements section? But we need to follow instruction: Prefer a few substantive sections over many thin ones. So we could have Requirements section that includes both the paragraph and the list. However we have separate headings for Requirements and Qualifications and Skills. We could put the paragraph under Qualifications, and the list under Requirements. Or we could put paragraph under About You (not allowed). Let's decide: Use Qualifications for the paragraph describing experience, and Requirements for the bullet list. Skills maybe not needed if we already covered in Requirements. But we could still include Skills heading for specific technologies (Spark, Python, Scala, Java, Kafka, etc.) but that would be thin. Better to omit Skills section if not needed. The instruction: include a section ONLY when the posting has real, specific content for it. So we can omit Skills if we don't have distinct content beyond what's in Requirements. However the Requirements list includes many skills; we could treat that as Requirements. The Qualifications paragraph is separate. So we will have Qualifications and Requirements sections.
Thus:
Qualifications
You are an experienced data engineer with deep expertise building and optimizing distributed data systems in cloud-native environments. You thrive solving complex scalability and performance challenges across high-volume data processing systems and enjoy operating in highly technical, fast-paced engineering environments. You bring strong hands-on engineering expertise across distributed computing, large-scale data processing, and infrastructure optimization while also helping guide technical direction and mentor engineers across the organization. Deep expertise with distributed data processing frameworks such as Apache Spark and Hadoop, particularly within AWS environments. Strong proficiency in Python (PySpark), Scala, Java, or other modern programming languages used for large-scale distributed processing. Experience building scalable ETL/ELT frameworks across both batch and streaming architectures. Experience with entity resolution, identity mapping, automatching, deduplication, or large-scale matching systems is strongly preferred. Strong understanding of distributed file formats including Apache Parquet and Apache AVRO. Experience with streaming technologies such as Kafka, Spark Streaming, or KSQL. Strong grasp of software engineering fundamentals including distributed systems, data structures, concurrency, and system design. Experience performing root cause analysis across large-scale distributed systems and complex data pipelines. Ability to write clean, maintainable, modular, and production-grade code. Experience improving performance, scalability, observability, and infrastructure efficiency within distributed systems. Strong communication and collaboration skills across both technical and non-technical stakeholders. Familiarity with modern development and infrastructure tooling including Git, CI/CD pipelines, Docker, Kubernetes, Terraform, Argo, Hudi, and JIRA.
Then Requirements:
Requirements
- 8+ years of experience building and maintaining large-scale distributed data systems and pipelines.
- Demonstrated technical leadership experience mentoring engineers and driving complex technical initiatives.
- Extensive experience with Apache Spark and AWS-based big data technologies including EMR, S3, and distributed compute environments.
- Strong coding experience in Python (PySpark), Scala, Java, or equivalent languages used for distributed processing systems.
- Experience optimizing large-scale Spark workloads for performance, scalability, and infrastructure cost efficiency.
- Experience with streaming and event-driven architectures using technologies such as Kafka or Spark Streaming.
- Experience with orchestration and lakehouse technologies such as Argo and Hudi or comparable platforms.
- Experience with containerization and infrastructure technologies such as Docker, Kubernetes, and Terraform.
- Experience working with relational or distributed databases such as PostgreSQL or Redshift.
- Proven ability to operate effectively within highly scalable, production-grade distributed systems.
- Experience working with healthcare, life sciences, Real World Evidence (RWE), or large-scale healthcare datasets is strongly preferred.
Then Benefits:
Benefits
- Full suite of health insurance options
- Generous paid time off
- Pre-planned company-wide wellness holidays
- Retirement options
- Health & charitable donation stipends
- Impactful Business Resource Groups
- Opportunity to work with leading biotech and life sciences companies in an innovative industry with a mission to improve healthcare around the globe
for prose. Do NOT repeat job title as heading. Do NOT put Job ID/requisition numbers as headings. Remove boilerplate. Preserve all substantive info. Include a section only when posting has real specific content for it. Avoid filler like "commensurate with experience". Pay and Schedule sections only if posting actually states them. Benefits only if posting actually states them. The posting includes compensation: "$190,000 to $220,000 per year, based on experience, in addition to stock options." Also includes benefits: "Full suite of health insurance options, in addition to generous paid time off Pre-planned company-wide wellness holidays Retirement options Health & charitable donation stipends Impactful Business Resource Groups Flexible work hours & the opportunity to work from anywhere The opportunity to work with leading biotech and life sciences companies in an innovative industry with a mission to improve healthcare around the globe". Also mentions "As part of H1’s hiring process, all candidates are required to participate in an in-person final interview. Depending on your location, this may require travel." That's maybe schedule? Not exactly schedule. There's no explicit schedule like "Full-time" etc. So we may not include Schedule section unless there is explicit schedule info. There's no schedule like "Full-time" or "Remote". There's "Flexible work hours & the opportunity to work from anywhere". That could be considered schedule? But schedule section is for work schedule (e.g., full-time, part-time, shift). The instruction: "never create Pay/Schedule/Benefits sections unless the posting actually states them." The posting does mention flexible work hours and remote work, which could be considered schedule. However it's not explicit like "Full-time". I'd consider that as schedule info. But we need to be careful: The instruction says "never create Pay/Schedule/Benefits sections unless the posting actually states them." The posting does state flexible work hours & opportunity to work from anywhere. That's schedule-related. So we could include a Schedule section with that info. However we also have Benefits section with health insurance, PTO, wellness holidays, retirement options, stipends, BRGs, flexible work hours, remote work, opportunity to work with leading biotech etc. Some of those overlap with schedule. But we can separate: Benefits: health insurance, PTO, wellness holidays, retirement options, health & charitable donation stipends, Impactful Business Resource Groups. Schedule: Flexible work hours & opportunity to work from anywhere. Also maybe "The opportunity to work with leading biotech..." is more about role, not schedule. We need to avoid duplication. Let's extract substantive content: - Mission statement: At H1, we believe access to the best healthcare information is a basic human right... (this is about company). Could be part of About the role? Usually "About the role" includes description of role and team. We'll put that in About the role. - Data Engineering team description: "Data Engineering is responsible for the development and delivery of our most important asset - our data...." This is about the team. - What You'll Do At H1: list of responsibilities. - About You: qualifications/requirements. - Requirements: list of requirements. - Compensation: pay. - Benefits: list. - Also mention: "As part of H1’s hiring process, all candidates are required to participate in an in-person final interview. Depending on your location, this may require travel." That's maybe not a section; could be placed in About the role or maybe as a note. But instruction: remove application instructions. This is about interview process, not application instructions. Might be considered metadata; but it's substantive. Could be placed in a short paragraph under About the role or maybe a separate section like "Interview Process". But we only have allowed headings list; we shouldn't create new headings not in list. So we should incorporate that info into About the role or maybe as a
under About the role. - Also mention "Visit h1.co to learn more about us." That's boilerplate? Could be removed as navigation? It's a link to learn more; maybe considered boilerplate. Probably remove. - Also mention "H1 OFFERS" then list benefits. We'll capture benefits. - Also mention "H1 is proud to be an equal opportunity employer..." that's EEO boilerplate; remove. - Also mention about AI tools in hiring process; that's also boilerplate; remove. Now structure:
About the role
At H1, we believe access to the best healthcare information is a basic human right. Our mission is to provide a platform that can optimally inform every doctor interaction globally. This promotes health equity and builds needed trust in healthcare systems. To accomplish this our teams harness the power of data and AI-technology to unlock groundbreaking medical insights and convert those insights into action that result in optimal patient outcomes and accelerates an equitable and inclusive drug development lifecycle. Data Engineering is responsible for the development and delivery of our most important asset - our data. Looking across thousands of data sources from across the globe, the data engineering team is responsible for making sense out of that data to create the world's most extensive and comprehensive knowledge base of healthcare stakeholders and the ecosystem they influence. It is our job to ensure that only accurate, normalized data flows through to our customers, and at a velocity that keeps up with the changes in the real world. As we rapidly expand the markets we serve and the breadth and depth of data we want to collect for our customers, the team must grow and scale to meet that demand. As part of H1’s hiring process, all candidates are required to participate in an in-person final interview. Depending on your location, this may require travel.
Then responsibilities:Responsibilities
- Lead the design, optimization, and scalability of distributed Spark/PySpark pipelines powering entity resolution and large-scale healthcare data processing.
- Own systems supporting automatching, identity mapping, grouping logic, deduplication, enrichment, and auto-approval workflows across healthcare provider and organization datasets.
- Build and maintain scalable processing frameworks for PubMed, clinical trial, ct.gov, conference, and other healthcare data sources.
- Drive infrastructure optimization initiatives focused on improving throughput, runtime, observability, and cloud compute cost efficiency.
- Partner closely with AI/ML teams to integrate matching and resolution models into EMERALD and improve matching precision and recall.
- Lead complex technical initiatives from architecture and design through deployment, monitoring, and long-term production support.
- Serve as a technical leader and mentor across the team through code reviews, technical guidance, and engineering best practices.
- Collaborate directly with Product and business stakeholders to align technical solutions with operational and customer needs.
- Support production operations, incident response, troubleshooting, and ongoing platform reliability.
Qualifications
You are an experienced data engineer with deep expertise building and optimizing distributed data systems in cloud-native environments. You thrive solving complex scalability and performance challenges across high-volume data processing systems and enjoy operating in highly technical, fast-paced engineering environments. You bring strong hands-on engineering expertise across distributed computing, large-scale data processing, and infrastructure optimization while also helping guide technical direction and mentor engineers across the organization. Deep expertise with distributed data processing frameworks such as Apache Spark and Hadoop, particularly within AWS environments. Strong proficiency in Python (PySpark), Scala, Java, or other modern programming languages used for large-scale distributed processing. Experience building scalable ETL/ELT frameworks across both batch and streaming architectures. Experience with entity resolution, identity mapping, automatching, deduplication, or large-scale matching systems is strongly preferred. Strong understanding of distributed file formats including Apache Parquet and Apache AVRO. Experience with streaming technologies such as Kafka, Spark Streaming, or KSQL. Strong grasp of software engineering fundamentals including distributed systems, data structures, concurrency, and system design. Experience performing root cause analysis across large-scale distributed systems and complex data pipelines. Ability to write clean, maintainable, modular, and production-grade code. Experience improving performance, scalability, observability, and infrastructure efficiency within distributed systems. Strong communication and collaboration skills across both technical and non-technical stakeholders. Familiarity with modern development and infrastructure tooling including Git, CI/CD pipelines, Docker, Kubernetes, Terraform, Argo, Hudi, and JIRA.
Then Requirements:Requirements
- 8+ years of experience building and maintaining large-scale distributed data systems and pipelines.
- Demonstrated technical leadership experience mentoring engineers and driving complex technical initiatives.
- Extensive experience with Apache Spark and AWS-based big data technologies including EMR, S3, and distributed compute environments.
- Strong coding experience in Python (PySpark), Scala, Java, or equivalent languages used for distributed processing systems.
- Experience optimizing large-scale Spark workloads for performance, scalability, and infrastructure cost efficiency.
- Experience with streaming and event-driven architectures using technologies such as Kafka or Spark Streaming.
- Experience with orchestration and lakehouse technologies such as Argo and Hudi or comparable platforms.
- Experience with containerization and infrastructure technologies such as Docker, Kubernetes, and Terraform.
- Experience working with relational or distributed databases such as PostgreSQL or Redshift.
- Proven ability to operate effectively within highly scalable, production-grade distributed systems.
- Experience working with healthcare, life sciences, Real World Evidence (RWE), or large-scale healthcare datasets is strongly preferred.
Benefits
- Full suite of health insurance options
- Generous paid time off
- Pre-planned company-wide wellness holidays
- Retirement options
- Health & charitable donation stipends
- Impactful Business Resource Groups
- Opportunity to work with leading biotech and life sciences companies in an innovative industry with a mission to improve healthcare around the globe