Senior Data Engineer
General Dynamics Information Technology · Fort Liberty, North Carolina, United States · 3 wk ago
$168k–$228k/yrFull-time
We need to output HTML fragment with sections: headings for sections like About the role, Responsibilities, Requirements, Qualifications, Skills, Benefits, Pay, Schedule. Only include sections where there is real specific content. Must not repeat job title. Must not include Job ID etc. Must preserve all substantive info, remove boilerplate (cookie/navigation/legal/EEO/equal-opportunity/scam-warning boilerplate, duplicated lines, application instructions). Also must not create Pay/Schedule/Benefits sections unless posting actually states them. The posting includes salary range and scheduled weekly hours, travel, telecommuting options, work location. So we have Pay (salary range) and Schedule (weekly hours). Benefits: there is a section about benefits (Total Rewards) describing medical, dental, vision, 401k, etc. So we can include Benefits section.
We need sections: About the role (maybe from "YOUR IMPACT", "OUR COMPANY", "JOB DESCRIPTION"? Actually "About the role" could be the description of impact and company. We'll include a section About the role with prose summarizing that.
Responsibilities: from "Job Duties Include (but Not Limited To):" list.
Requirements: maybe "Required Skills & Experience:" list.
Qualifications: maybe "WHAT YOU’LL NEED TO SUCCEED: Education & Certifications:" plus Security Clearance and US Citizenship.
Skills: maybe combine with Required Skills & Experience? But we already have Requirements. Could separate Skills as a section listing technical skills. However we must avoid thin sections. Perhaps combine Requirements and Skills into one? But instructions: Use headings for section headings (e.g. About the role, Responsibilities, Requirements, Qualifications, Skills, Benefits, Pay, Schedule). We can include multiple if they have substantive content. We have Responsibilities, Requirements (skills & experience), Qualifications (education, clearance, citizenship), Benefits, Pay, Schedule. Also maybe About the role.
We must not include filler like "commensurate with experience". We have specific salary range, weekly hours, travel, telecommuting, location.
We must not include duplicated lines. Remove EEO boilerplate etc.
Let's extract content.
From posting:
- YOUR IMPACT: "Own your opportunity to work with the largest government agency in the nation. Make an impact by advancing the Department of War’s mission to keep our country safe and secure."
- OUR COMPANY: "Iron EagleX (IEX), a wholly owned subsidiary of General Dynamics Information Technology (GDIT), delivers agile IT and Intelligence solutions. Combining small-team flexibility with global scale, IEX leverages emerging technologies to provide innovative, user-focused solutions that empower organizations and end users to operate smarter, faster, and more securely in dynamic environments."
- JOB DESCRIPTION: "We are seeking a Data Engineer SME with a strong ETL emphasis to design, build, and maintain data collection, processing, and integration pipelines that centralize and structure data from various web-based applications. You will work closely with software developers, data analysts, and product teams to ensure seamless data flow, high-quality data storage, and efficient retrieval methods. This role is ideal for someone passionate about data-driven decision-making and optimizing data aggregation processes while exploring and identifying right-sized technologies and computing resources. This position focuses on wrangling disparate data sources into unified, consumable data APIs for internal and external stakeholders. This is a hybrid position (3 days on site) in Fort Bragg, NC."
- Job Duties Include (but Not Limited To): list:
1. Design, develop, and implement scalable data pipelines and ETL processes using Apache Airflow.
2. Develop messaging solutions utilizing Kafka to support real-time data streaming and event-driven architectures.
3. Build and maintain high-performance data retrieval solutions using ElasticSearch/OpenSearch.
4. Implement and optimize Python-based data processing solutions.
5. Integrate batch and streaming data processing techniques to enhance data availability and accessibility.
6. Ensure adherence to security and compliance requirements when working with classified data.
7. Work closely with cross-functional teams to define data strategies and develop technical solutions aligned with mission objectives.
8. Design, implement, and maintain robust data engineering solutions serving data to the backend of production user facing applications.
9. Deploy and manage cloud-based infrastructure to support scalable and resilient data solutions, including AWS S3.
10. Optimize data storage, retrieval, and processing efficiency, focusing on big data and the use of Parquet files.
11. Utilize Trino for distributed SQL query processing.
12. Leverage OpenMetaData for data discovery and governance.
13. Implement APIs using FastAPI for streamlined data interaction.
14. Ensure proper implementation of Data Governance requirements and adherence to common PKI security practices to appropriately protect data.
15. Leverage IC repositories of user attributes to provide authorized access to data and ensure proper protections are applied.
- Required Skills & Experience: list:
- Extensive experience with Apache Airflow for workflow orchestration.
- Strong programming skills in Python.
- Experience using SQL compliant database solutions such as Postgres.
- Familiarity with Trino for enhanced SQL query processing in large datasets.
- Experience with ElasticSearch/OpenSearch for data indexing and search functionalities.
- Understanding of vector databases, embedding models, and vector search for AI applications.
- Expertise in event-driven architecture and microservices development.
- Hands-on experience with cloud services, particularly AWS S3 and MinIO, including data storage and compute resources.
- Strong knowledge of Reggo and its application within data environments. (Note: "Reggo" maybe typo but keep)
- Understanding of big data technologies and file formats like Parquet.
- Expertise in data pipeline orchestration and workflow automation.
- Working knowledge of Linux environments and database optimization techniques.
- Strong understanding of version control with Git.
- Familiarity with OpenMetaData for data management.
- Experience implementing FastAPI for building scalable APIs.
- WHAT YOU’LL NEED TO SUCCEED: Education & Certifications: Bachelor's degree in Computer Science, Information Systems, Engineering, or a related field (or equivalent experience). Advanced degrees are a plus. Security Clearance: An active TS/SCI security clearance is REQUIRED, and candidates must have or be willing to obtain a CI Poly. Candidates without this clearance will not be considered. Due to US Government Contract Requirements, only US Citizens are eligible for this role.
- IEX/GDIT IS YOUR PLACE: bullet points: Growth: AI-powered career tool... Support: internal mobility team... Rewards: Comprehensive benefits and wellness packages, 401K with company match, competitive pay and paid time off Community: Award-winning culture of innovation and a military-friendly workplace
- OWN YOUR OPPORTUNITY: (maybe not needed)
- Salary range: "The likely salary range for this position is $168,300 - $227,700. This is not, however, a guarantee of compensation or salary. Rather, salary will be set based on experience, geographic location and possibly contractual requirements and could fall outside of this range."
- Scheduled Weekly Hours: 40
- Travel Required: Less than 10%
- Telecommuting Options: Hybrid Work Location: USA NC Fort Bragg
- Additional Work Locations: (maybe none)
- Total Rewards At GDIT: benefits description.
We need Benefits section: include medical plan options, dental, vision, 401k, flex work weeks, paid time off, disability, life, AD&D, personal accident, critical illness, business travel insurance, etc.
We should not include boilerplate like Equal Opportunity Employer etc. Remove those lines.
We also need to remove duplicated lines: there are duplicate EEO statements at end; we will omit.
Now craft HTML.
We'll have:
for sections like About the role, Responsibilities, Requirements, Qualifications, Skills, Benefits, Pay, Schedule. Only include sections where there is real specific content. Must not repeat job title. Must not include Job ID etc. Must preserve all substantive info, remove boilerplate (cookie/navigation/legal/EEO/equal-opportunity/scam-warning boilerplate, duplicated lines, application instructions). Also must not create Pay/Schedule/Benefits sections unless posting actually states them. The posting includes salary range and scheduled weekly hours, travel, telecommuting options, work location. So we have Pay (salary range) and Schedule (weekly hours). Benefits: there is a section about benefits (Total Rewards) describing medical, dental, vision, 401k, etc. So we can include Benefits section.
We need sections: About the role (maybe from "YOUR IMPACT", "OUR COMPANY", "JOB DESCRIPTION"? Actually "About the role" could be the description of impact and company. We'll include a section About the role with prose summarizing that.
Responsibilities: from "Job Duties Include (but Not Limited To):" list.
Requirements: maybe "Required Skills & Experience:" list.
Qualifications: maybe "WHAT YOU’LL NEED TO SUCCEED: Education & Certifications:" plus Security Clearance and US Citizenship.
Skills: maybe combine with Required Skills & Experience? But we already have Requirements. Could separate Skills as a section listing technical skills. However we must avoid thin sections. Perhaps combine Requirements and Skills into one? But instructions: Use headings for section headings (e.g. About the role, Responsibilities, Requirements, Qualifications, Skills, Benefits, Pay, Schedule). We can include multiple if they have substantive content. We have Responsibilities, Requirements (skills & experience), Qualifications (education, clearance, citizenship), Benefits, Pay, Schedule. Also maybe About the role.
We must not include filler like "commensurate with experience". We have specific salary range, weekly hours, travel, telecommuting, location.
We must not include duplicated lines. Remove EEO boilerplate etc.
Let's extract content.
From posting:
- YOUR IMPACT: "Own your opportunity to work with the largest government agency in the nation. Make an impact by advancing the Department of War’s mission to keep our country safe and secure."
- OUR COMPANY: "Iron EagleX (IEX), a wholly owned subsidiary of General Dynamics Information Technology (GDIT), delivers agile IT and Intelligence solutions. Combining small-team flexibility with global scale, IEX leverages emerging technologies to provide innovative, user-focused solutions that empower organizations and end users to operate smarter, faster, and more securely in dynamic environments."
- JOB DESCRIPTION: "We are seeking a Data Engineer SME with a strong ETL emphasis to design, build, and maintain data collection, processing, and integration pipelines that centralize and structure data from various web-based applications. You will work closely with software developers, data analysts, and product teams to ensure seamless data flow, high-quality data storage, and efficient retrieval methods. This role is ideal for someone passionate about data-driven decision-making and optimizing data aggregation processes while exploring and identifying right-sized technologies and computing resources. This position focuses on wrangling disparate data sources into unified, consumable data APIs for internal and external stakeholders. This is a hybrid position (3 days on site) in Fort Bragg, NC."
- Job Duties Include (but Not Limited To): list:
1. Design, develop, and implement scalable data pipelines and ETL processes using Apache Airflow.
2. Develop messaging solutions utilizing Kafka to support real-time data streaming and event-driven architectures.
3. Build and maintain high-performance data retrieval solutions using ElasticSearch/OpenSearch.
4. Implement and optimize Python-based data processing solutions.
5. Integrate batch and streaming data processing techniques to enhance data availability and accessibility.
6. Ensure adherence to security and compliance requirements when working with classified data.
7. Work closely with cross-functional teams to define data strategies and develop technical solutions aligned with mission objectives.
8. Design, implement, and maintain robust data engineering solutions serving data to the backend of production user facing applications.
9. Deploy and manage cloud-based infrastructure to support scalable and resilient data solutions, including AWS S3.
10. Optimize data storage, retrieval, and processing efficiency, focusing on big data and the use of Parquet files.
11. Utilize Trino for distributed SQL query processing.
12. Leverage OpenMetaData for data discovery and governance.
13. Implement APIs using FastAPI for streamlined data interaction.
14. Ensure proper implementation of Data Governance requirements and adherence to common PKI security practices to appropriately protect data.
15. Leverage IC repositories of user attributes to provide authorized access to data and ensure proper protections are applied.
- Required Skills & Experience: list:
- Extensive experience with Apache Airflow for workflow orchestration.
- Strong programming skills in Python.
- Experience using SQL compliant database solutions such as Postgres.
- Familiarity with Trino for enhanced SQL query processing in large datasets.
- Experience with ElasticSearch/OpenSearch for data indexing and search functionalities.
- Understanding of vector databases, embedding models, and vector search for AI applications.
- Expertise in event-driven architecture and microservices development.
- Hands-on experience with cloud services, particularly AWS S3 and MinIO, including data storage and compute resources.
- Strong knowledge of Reggo and its application within data environments. (Note: "Reggo" maybe typo but keep)
- Understanding of big data technologies and file formats like Parquet.
- Expertise in data pipeline orchestration and workflow automation.
- Working knowledge of Linux environments and database optimization techniques.
- Strong understanding of version control with Git.
- Familiarity with OpenMetaData for data management.
- Experience implementing FastAPI for building scalable APIs.
- WHAT YOU’LL NEED TO SUCCEED: Education & Certifications: Bachelor's degree in Computer Science, Information Systems, Engineering, or a related field (or equivalent experience). Advanced degrees are a plus. Security Clearance: An active TS/SCI security clearance is REQUIRED, and candidates must have or be willing to obtain a CI Poly. Candidates without this clearance will not be considered. Due to US Government Contract Requirements, only US Citizens are eligible for this role.
- IEX/GDIT IS YOUR PLACE: bullet points: Growth: AI-powered career tool... Support: internal mobility team... Rewards: Comprehensive benefits and wellness packages, 401K with company match, competitive pay and paid time off Community: Award-winning culture of innovation and a military-friendly workplace
- OWN YOUR OPPORTUNITY: (maybe not needed)
- Salary range: "The likely salary range for this position is $168,300 - $227,700. This is not, however, a guarantee of compensation or salary. Rather, salary will be set based on experience, geographic location and possibly contractual requirements and could fall outside of this range."
- Scheduled Weekly Hours: 40
- Travel Required: Less than 10%
- Telecommuting Options: Hybrid Work Location: USA NC Fort Bragg
- Additional Work Locations: (maybe none)
- Total Rewards At GDIT: benefits description.
We need Benefits section: include medical plan options, dental, vision, 401k, flex work weeks, paid time off, disability, life, AD&D, personal accident, critical illness, business travel insurance, etc.
We should not include boilerplate like Equal Opportunity Employer etc. Remove those lines.
We also need to remove duplicated lines: there are duplicate EEO statements at end; we will omit.
Now craft HTML.
We'll have:
About the role
Own your opportunity to work with the largest government agency in the nation. Make an impact by advancing the Department of War’s mission to keep our country safe and secure. Iron EagleX (IEX), a wholly owned subsidiary of General Dynamics Information Technology (GDIT), delivers agile IT and Intelligence solutions. Combining small-team flexibility with global scale, IEX leverages emerging technologies to provide innovative, user-focused solutions that empower organizations and end users to operate smarter, faster, and more securely in dynamic environments. We are seeking a Data Engineer SME with a strong ETL emphasis to design, build, and maintain data collection, processing, and integration pipelines that centralize and structure data from various web-based applications. You will work closely with software developers, data analysts, and product teams to ensure seamless data flow, high-quality data storage, and efficient retrieval methods. This role is ideal for someone passionate about data-driven decision-making and optimizing data aggregation processes while exploring and identifying right-sized technologies and computing resources. This position focuses on wrangling disparate data sources into unified, consumable data APIs for internal and external stakeholders. This is a hybrid position (3 days on site) in Fort Bragg, NC.
Responsibilities
- Design, develop, and implement scalable data pipelines and ETL processes using Apache Airflow.
- Develop messaging solutions utilizing Kafka to support real-time data streaming and event-driven architectures.
- Build and maintain high-performance data retrieval solutions using ElasticSearch/OpenSearch.
- Implement and optimize Python-based data processing solutions.
- Integrate batch and streaming data processing techniques to enhance data availability and accessibility.
- Ensure adherence to security and compliance requirements when working with classified data.
- Work closely with cross-functional teams to define data strategies and develop technical solutions aligned with mission objectives.
- Design, implement, and maintain robust data engineering solutions serving data to the backend of production user facing applications.
- Deploy and manage cloud-based infrastructure to support scalable and resilient data solutions, including AWS S3.
- Optimize data storage, retrieval, and processing efficiency, focusing on big data and the use of Parquet files.
- Utilize Trino for distributed SQL query processing.
- Leverage OpenMetaData for data discovery and governance.
- Implement APIs using FastAPI for streamlined data interaction.
- Ensure proper implementation of Data Governance requirements and adherence to common PKI security practices to appropriately protect data.
- Leverage IC repositories of user attributes to provide authorized access to data and ensure proper protections are applied.
Requirements
- Extensive experience with Apache Airflow for workflow orchestration.
- Strong programming skills in Python.
- Experience using SQL compliant database solutions such as Postgres.
- Familiarity with Trino for enhanced SQL query processing in large datasets.
- Experience with ElasticSearch/OpenSearch for data indexing and search functionalities.
- Understanding of vector databases, embedding models, and vector search for AI applications.
- Expertise in event-driven architecture and microservices development.
- Hands-on experience with cloud services, particularly AWS S3 and MinIO, including data storage and compute resources.
- Strong knowledge of Reggo and its application within data environments.
- Understanding of big data technologies and file formats like Parquet.
- Expertise in data pipeline orchestration and workflow automation.
- Working knowledge of Linux environments and database optimization techniques.
- Strong understanding of version control with Git.
- Familiarity with OpenMetaData for data management.
- Experience implementing FastAPI for building scalable APIs.
Qualifications
- Bachelor's degree in Computer Science, Information Systems, Engineering, or a related field (or equivalent experience). Advanced degrees are a plus.
- An active TS/SCI security clearance is REQUIRED, and candidates must have or be willing to obtain a CI Poly.
- Due to US Government Contract Requirements, only US Citizens are eligible for this role.
Benefits
- Comprehensive benefits and wellness packages.
- 401(k) with company match.