Data Engineer

Responsibilities

• Identify, design, and implement internal process improvements, including re-designing infrastructure for greater scalability, optimizing data delivery, and automating manual processes.
• Develop and design data pipelines to support an end-to-end solution.
• Develop and maintain artifacts (e.g., schemas, data dictionaries, and transforms related to ETL processes).
• Integrate data pipelines with AWS cloud services to extract meaningful insights.
• Work with stakeholders to support their data infrastructure needs and assist with data-related technical issues.
• Design and develop robust and functional data flows to support raw data and expected data.
• Provide Tier 3 technical support for deployed applications and dataflows.
• Define and communicate a clear product vision for the client’s software products, aligning with user needs and business objectives.
• Create and manage product roadmaps that reflect both innovation and growth strategies.
• Partner with a government product owner and a product team of 7 - 8 full-time employees.
• Collaborate with the rest of the data engineering team to design and launch new features.
• Coordinate and document dataflows, capabilities, etc.
• Occasionally (as needed) support for off-hours deployment, such as evenings or weekends.

Requirements

• An active Top Secret SCI clearance.
• A Bachelor's degree or equivalent practical experience.
• Expertise in distributed computing frameworks to handle large-scale data processing.
• Familiarity with working with Amazon Web Services (AWS) or any other cloud environments.
• Working experience with datastores like PostgreSQL, S3, Redshift, MongoDB/DynamoDB, Redis, Elasticsearch/OpenSearch, and SQL.
• Proficient in utilizing Python with key libraries like pandas and PySpark, NiFi, Airflow, AWS Lambda, or similar technologies.
• Working knowledge of software platforms and services, such as Docker, Kubernetes, JMS/SQS, SNS, and Kafka.
• Familiar with Linux/Unix server environments.
• Experience with Agile development methodology.
• Publishing and/or presenting design reports.
• Coordinating with other team members to reach project milestones and deadlines.
• Working knowledge of Collaboration tools, such as Jira and Confluence.

Preferred Qualifications

• A Master’s degree or equivalent experience in a related field.
• Familiarity and experience with the Intelligence Community and the intel cycle.
• Experience with cloud message APIs and usage of push notifications.
• Keen interest in learning and using the latest software tools, methods, and technologies to solve real-world problem sets vital to national security.
• Working knowledge of public keys and digital certificates.
• Experience with DevOps environments.
• Expertise in various COTS, GOTS, and open source tools that support the development of data integration and visualization applications.
• Experience with cloud message APIs and usage of push notifications.
• Specialization in object-oriented programming languages, scripting, and databases.