Senior Data Engineer

We need to output HTML fragment with sections: About the role (maybe from Job Summary), Responsibilities, Requirements (or Required Skills), Qualifications (Preferred Qualifications), Skills (maybe Technical Environment?), Benefits, Pay, Schedule only if present. The posting does not mention Benefits, Pay, Schedule specifics. So we should not include those sections. Also we should not repeat job title. We need to include sections only if real content. We have Job Summary paragraph. Could be "About the role". Use

About the role

then

with summary. Responsibilities: list bullet points from "Key Responsibilities". Use

• . Required Skills: could be "Requirements" or "Skills". The instruction says use headings like Requirements, Qualifications, Skills, Benefits, Pay, Schedule. We have Required Skills and Preferred Qualifications. So we can have

  Requirements

  for Required Skills, and

  Preferred Qualifications

  for Preferred Qualifications. Also maybe a

  Skills

  for Technical Environment? But we already have Requirements covering many skills; Technical Environment could be considered Skills. However we should avoid thin sections. The Technical Environment lists languages, big data, cloud, etc. Could be merged into Requirements or Skills. Since we already have Requirements (Required Skills) and Preferred Qualifications, we could add a Skills section for Technical Environment if it adds value beyond those. But the Technical Environment is essentially a summary of tools; it's somewhat overlapping with Required Skills. Might be considered redundant. The instruction: Prefer a few substantive sections over many thin ones. So maybe we omit separate Skills section and keep Requirements and Preferred Qualifications. We also have Work Arrangement: "This is a Full-Time Hybrid position...". That could go into a short paragraph under About the role or maybe a separate section? There's no heading for Work Arrangement; we could incorporate into About the role paragraph. We must not include boilerplate like Job Type, Work Model, Experience, Employment lines. Those are metadata; we can drop them or fold into short p. The instruction: do NOT put Job ID / requisition numbers as headings — fold minor metadata into a short

  or drop it. So we could include a short p at top with "Job Type: Full-Time, Work Model: Hybrid, Experience: 8+ Years". But maybe better to drop as it's minor. However we need to preserve ALL substantive information; is that substantive? It's about job type, work model, experience. Might be considered substantive but we can incorporate into About the role paragraph. The instruction: "Preserve ALL substantive information; do NOT invent or summarize away detail." So we need to keep that info somewhere. We can include a short

  after the heading About the role that includes those details. Thus structure:

  About the role

  Full-Time Hybrid position requiring 8+ years of experience. We are looking for an experienced Senior Data Engineer with 8+ years of hands-on experience designing, developing, and maintaining scalable data platforms and data pipelines. The ideal candidate will have strong expertise in Python, SQL, ETL/ELT, cloud platforms, data warehousing, and distributed data processing. The candidate will work closely with Data Scientists, Software Engineers, Business Analysts, and other stakeholders to build reliable, high-performance data solutions and enable data-driven decision-making.

  Then Responsibilities. Then Requirements (Required Skills). Then Preferred Qualifications. We could also include a short p for "Work Arrangement" but we already included Full-Time Hybrid. We must not include Benefits, Pay, Schedule. Now ensure we preserve all bullet points from Responsibilities and Required Skills and Preferred Qualifications. Let's extract Responsibilities list: - Design, develop, and maintain scalable ETL/ELT data pipelines. - Build and optimize data processing workflows using Python and SQL. - Develop data ingestion solutions from databases, APIs, files, and other sources. - Design and maintain enterprise data warehouses, data lakes, and lakehouse architectures. - Work with large datasets using distributed processing technologies such as Apache Spark/PySpark. - Develop data models and optimize complex SQL queries for performance. - Implement data quality, validation, monitoring, and error-handling frameworks. - Integrate data from structured and unstructured sources. - Build and maintain batch and real-time/streaming data pipelines. - Work with cloud data platforms such as AWS, Azure, or GCP. - Implement CI/CD and DevOps practices for data engineering workflows. - Collaborate with cross-functional teams to understand business and technical requirements. - Troubleshoot pipeline failures, performance issues, and data quality problems. - Ensure data security, governance, scalability, and compliance. - Mentor junior and mid-level data engineers and contribute to technical architecture decisions. - Document data pipelines, architectures, processes, and technical solutions. Now Required Skills list: - 8+ years of professional experience in Data Engineering or related roles. - Strong programming experience with Python. - Advanced SQL skills with experience in query optimization. - Strong experience with ETL/ELT development and data pipeline architecture. - Hands-on experience with Apache Spark / PySpark. - Experience with cloud platforms such as: - AWS – S3, Glue, Redshift, EMR, Lambda - Azure – Data Factory, Databricks, Synapse, ADLS - GCP – BigQuery, Dataflow, Cloud Storage - Strong understanding of data warehousing and dimensional data modeling. - Experience with relational databases such as SQL Server, PostgreSQL, Oracle, or MySQL. - Experience with Databricks and/or Snowflake is highly desirable. - Knowledge of Apache Kafka or other streaming technologies. - Experience with workflow orchestration tools such as Airflow, Azure Data Factory, or AWS Glue. - Familiarity with Git, CI/CD, Docker, and DevOps practices. - Strong understanding of data quality, governance, security, and metadata management. Preferred Qualifications: - Bachelor's or Master's degree in Computer Science, Engineering, Information Technology, or a related field. - Experience designing enterprise-scale data architectures. - Experience with real-time data processing and streaming. - Experience with Terraform or Infrastructure as Code. - Familiarity with Kubernetes and containerized workloads. - Experience working in Agile/Scrum environments. - Strong communication, analytical, and problem-solving skills. Technical Environment list (maybe we can incorporate into Requirements or Skills but we already have many). Could add a Skills section but risk thin. However we could add a

  Technical Environment

  with a

  summarizing. But the instruction: Prefer a few substantive sections over many thin ones. The Technical Environment is somewhat redundant with Required Skills. Might be better to omit. But we need to preserve ALL substantive information. The Technical Environment includes some items not explicitly in Required Skills: Scala/Java, Hadoop, Snowflake, Redshift, Synapse, BigQuery (some already), etc. Some are covered but not all. For example, Scala/Java, Hadoop, Snowflake (mentioned as highly desirable), etc. Some are duplicates. To be safe, we could include a Skills section that captures the Technical Environment, but we must avoid thin sections. The Technical Environment list is fairly substantive; we could make a

  Skills

  with a
  • list. But we already have Requirements (Required Skills) which includes many of those. However the Technical Environment includes Scala/Java, Hadoop, etc. Not in Required Skills. So we need to preserve that. Thus we could have:

    Requirements

    (Required Skills)

    Preferred Qualifications

    Technical Environment

    (list) That's three sections plus About the role and Responsibilities = 5 sections. That's okay. Now ensure we don't create sections with filler like "commensurate with experience". Not present. Now output HTML fragment only. Let's craft. We'll start with

    About the role

    then

    containing job type etc and summary. We need to be careful to not repeat job title. Let's write:

    About the role

    Full-Time Hybrid position requiring 8+ years of experience. We are looking for an experienced Senior Data Engineer with 8+ years of hands-on experience designing, developing, and maintaining scalable data platforms and data pipelines. The ideal candidate will have strong expertise in Python, SQL, ETL/ELT, cloud platforms, data warehousing, and distributed data processing. The candidate will work closely with Data Scientists, Software Engineers, Business Analysts, and other stakeholders to build reliable, high-performance data solutions and enable data-driven decision-making.

    Then Responsibilities.

    Responsibilities

    • Design, develop, and maintain scalable ETL/ELT data pipelines.
    • Build and optimize data processing workflows using Python and SQL.
    • Develop data ingestion solutions from databases, APIs, files, and other sources.
    • Design and maintain enterprise data warehouses, data lakes, and lakehouse architectures.
    • Work with large datasets using distributed processing technologies such as Apache Spark/PySpark.
    • Develop data models and optimize complex SQL queries for performance.
    • Implement data quality, validation, monitoring, and error-handling frameworks.
    • Integrate data from structured and unstructured sources.
    • Build and maintain batch and real-time/streaming data pipelines.
    • Work with cloud data platforms such as AWS, Azure, or GCP.
    • Implement CI/CD and DevOps practices for data engineering workflows.
    • Collaborate with cross-functional teams to understand business and technical requirements.
    • Troubleshoot pipeline failures, performance issues, and data quality problems.
    • Ensure data security, governance, scalability, and compliance.
    • Mentor junior and mid-level data engineers and contribute to technical architecture decisions.
    • Document data pipelines, architectures, processes, and technical solutions.
    Then Requirements.

    Requirements

    • 8+ years of professional experience in Data Engineering or related roles.
    • Strong programming experience with Python.
    • Advanced SQL skills with experience in query optimization.
    • Strong experience with ETL/ELT development and data pipeline architecture.
    • Hands-on experience with Apache Spark / PySpark.
    • Experience with cloud platforms such as:
      • AWS – S3, Glue, Redshift, EMR, Lambda
      • Azure – Data Factory, Databricks, Synapse, ADLS
      • GCP – BigQuery, Dataflow, Cloud Storage
    • Strong understanding of data warehousing and dimensional data modeling.
    • Experience with relational databases such as SQL Server, PostgreSQL, Oracle, or MySQL.
    • Experience with Databricks and/or Snowflake is highly desirable.
    • Knowledge of Apache Kafka or other streaming technologies.
    • Experience with workflow orchestration tools such as Airflow, Azure Data Factory, or AWS Glue.
    • Familiarity with Git, CI/CD, Docker, and DevOps practices.
    • Strong understanding of data quality, governance, security, and metadata management.
    But note nested ul inside li may be okay but could be messy. Better to keep as separate li with dash? Could flatten: "Experience with cloud platforms such as AWS (S3, Glue, Redshift, EMR, Lambda), Azure (Data Factory, Databricks, Synapse, ADLS), GCP (BigQuery, Dataflow, Cloud Storage)." That's a single li. Let's do that. Similarly for other lists. Now Preferred Qualifications.

    Preferred Qualifications

    • Bachelor's or Master's degree in Computer Science, Engineering, Information Technology, or a related field.
    • Experience