Data Engineering Lead
Position Overview
The Data Engineering Lead is responsible for designing and implementing modern, scalable data architectures to support migration of legacy, file-based analytical systems to AWS Cloud Native environments.
Key Responsibilities
Legacy Data Discovery & Data Model Transformation
- Analyze legacy storage models and design target-state data models aligned to AWS Cloud Native architecture
- Replace file-driven batch dependencies with API-based ingestion, event-driven workflows, and database-backed storage (e.g., Aurora/Postgres)
- Define canonical data schemas and transformation standards
- Optimize data storage for performance, scalability, and cost efficiency
- Support serverless and containerized data processing architectures
Expert Python-Based Data Engineering
- Develop advanced Python-based data transformation and validation pipelines
- Implement modular, reusable data processing components
- Optimize large-scale data manipulation for distributed execution
- Develop high-performance ETL/ELT frameworks
- Embed automated validation checks directly into data pipelines
Data Accuracy, Validation & Visibility
- Design and implement automated data validation frameworks to ensure functional equivalence, record-level and aggregate-level consistency, and downstream compatibility across subsystems
- Develop dashboards and reporting mechanisms providing data accuracy metrics, pipeline health indicators, variance detection summaries, and transparency into data transformation impacts
- Support regression validation through golden datasets and automated comparisons
Data Coordination
- Coordinate with Senior Developers and Requirements Engineers to align data models with application modernization
- Ensure upstream/downstream data contract stability
- Support orchestration of gated workflows through automated triggers rather than manual file exchanges
- Collaborate across workstreams to establish shared data standards
DevSecOps & Governance Alignment
- Integrate data pipelines into CI/CD frameworks
- Support infrastructure-as-code alignment (Terraform/CloudFormation collaboration)
- Ensure compliance with security controls (IAM, encryption, key management)
- Produce documentation supporting architecture review boards, interface control documents, data flow diagrams, and ATO-related data validation evidence
Requirements
8+ years of experience in data engineering or data architecture
Expert-level proficiency in Python for data engineering
Demonstrated experience transforming legacy file-based systems into cloud-native data architectures
Experience developing data models for high-volume, data-intensive applications
Deep experience with AWS data services (Glue, Lambda, S3, Aurora/Postgres, EventBridge, etc.)
Experience designing scalable ETL/ELT pipelines
Experience building analytical dashboards (e.g., QuickSight or equivalent)
Experience implementing automated data validation and quality controls
Experience working in Agile Scrum Teams
U.S. Citizenship required
Preferred Qualifications
Experience modernizing SAS-based data environments
Experience supporting system-of-systems integration programs
Experience implementing data lineage and metadata management
Experience operating in regulated or federal environments
Key Competencies
Systems-level thinking across data ecosystems
Strong schema design and normalization expertise
Data accuracy and integrity focus
Automation-first mindset
Cross-workstream coordination capability
Benefits
401(k) with matching and 100% Vested
Health Insurance - 3 plans to select from
Dental insurance
Vision Insurance
Health savings account
Life insurance
Short Term Disability
Long Term Disability
AD&D
Paid time off
Professional development assistance
Training
Tuition reimbursement
Flexible schedule
Flexible spending account
Referral program
Paid Legal Plan