Data Engineer - Mid Level

Responsibilities

• Designs, implements, and maintains large-scale ETL pipelines and ingestion architecture for collecting, retrieving, normalizing, validating, and processing intelligence data.
• Buils and maintains distributed, event-driven architecture using cloud-native services such as AWS Lambda, EventBridge, SQS, RDS/PostgreSQL, and serverless computing platforms.
• Develops backend services, APIs, and worker-based systems for parsing, transforming, enriching, and publishing structured data.
• Drafts database schemas and indexes, and manages data integrity and performance of PostgreSQL database environment.
• Creates automated validation, deduplication, and quality assurance processes to ensure accuracy, consistency, and reliability of ingested data.
• Implements monitoring, alerting, and logging mechanisms to detect pipeline failure and processing issues in a distributed environment.
• Participates in collaboration with engineers, analytics, and mission stakeholders to determine data processing requirements.
• Investigates, analyzes, and implements new and innovative technologies to enhance the scalability and automation of cloud-native data platforms.
• Implements best practices of software development and cloud infrastructure in accordance with mission and security (RMF) guidelines.
• Supports CI/CD (GitLab CI/CD, or similar) pipelines, automation of infrastructure (Terraform, Terragrunt), and deployment processes for cloud-native applications and serverless services.
• Documents and creates technical architecture and operational procedures.

Qualifications

• 4-10 years of experience in Data Engineering, Backend Engineering, and DevOps.
• Bachelor of Science or Administration in related field.
• AWS Certified Cloud Practitioner and/or AWS Certified Developer Associate.
• CompTIA Security+ certification.