Data Engineer – Analytic Platform & Data Pipelines

About the role

3GIMBALS is seeking a Data Engineer to design, build, and maintain the data pipelines and infrastructure that power our unclassified PAI/CAI-based analytic platform. This role is responsible for ingesting, transforming, and curating large volumes of structured and unstructured data from diverse open and commercial sources; building resilient, automated ETL/ELT workflows; and ensuring data is high-quality, well-governed, and analysis-ready for the downstream analytics, knowledge graph, and modeling teams. The ideal candidate is comfortable working with messy, multi-source data at scale within secure development environments. This role is central to the platform: the data engineering team delivers the clean, trustworthy, well-documented data that every analytic, knowledge graph, and risk-modeling capability depends on.

Responsibilities

• Design and build scalable batch and streaming pipelines to ingest structured and unstructured data from PAI/CAI sources, APIs, and third-party feeds
• Develop ETL/ELT workflows to normalize, enrich, and transform heterogeneous data into standardized schemas
• Build and maintain automated ingestion connectors for web, document, geospatial, and tabular data sources
• Manage data orchestration and scheduling using tools such as Airflow, Dagster, or Prefect
• Design and maintain data models, schemas, and storage layers across relational, NoSQL, and object stores
• Build and maintain data lakes/lakehouses and curated, analysis-ready data marts
• Optimize partitioning, indexing, and query performance for large datasets
• Support entity resolution and data linking in coordination with the knowledge graph and modeling teams
• Implement data validation, quality checks, and monitoring across pipelines
• Establish data lineage, cataloging, and metadata management
• Enforce data governance, provenance tracking, and source attribution appropriate for PAI/CAI data
• Document datasets, schemas, and pipeline logic for downstream consumers
• Ensure pipelines and data stores meet security requirements for operation in sensitive environments
• Implement encryption, access control, and secure data-handling practices
• Support Authority to Operate (ATO) processes and compliance frameworks

Requirements

• 4+ years of data engineering experience building and operating production data pipelines
• Strong programming skills in Python and SQL (Scala or Java a plus)
• Experience with distributed data processing frameworks (Spark, Dask, or similar)
• Hands-on experience with workflow orchestration tools (Airflow, Dagster, Prefect)
• Proficiency with relational and NoSQL databases (PostgreSQL, MongoDB, Elasticsearch, etc.)
• Experience with cloud data platforms and services (AWS, Azure, or GCP)

Qualifications

• Active security clearance or ability to obtain one
• Experience in government, defense, or intelligence contracting environments
• Familiarity with PAI/CAI (publicly and commercially available information) data sources
• Experience with geospatial data processing (PostGIS, GDAL, or similar)
• Knowledge of graph data structures and preparing data for knowledge graphs
• Experience with streaming platforms (Kafka, Kinesis)
• Familiarity with federal compliance frameworks (FedRAMP, FISMA, NIST 800-53)

Skills

Technical Environment: Languages: Python, SQL (Scala/Java a plus); Processing: Spark, Airflow/Dagster/Prefect, streaming frameworks; Storage: PostgreSQL, Elasticsearch, object storage / data lake, Parquet; Infrastructure: Docker, Kubernetes, cloud platforms (AWS GovCloud, Azure Government); Security: Encryption at rest and in transit, RBAC, secure data handling.