Software Engineer – Data Pipeline (Remote)

Role: Software Engineer – Data Pipeline (Remote)Location: Remote (Work from Anywhere)Job Type: Full-TimePayout: $10 Role Overview:The role involves building the technical implementation of a system that captures GitHub Copilot CLI agent traces across developer machines, scrubs sensitive content centrally in Azure, and serves a sanitized, queryable dataset for review. You will own the pipeline end‑to‑end, from the endpoint capture script to the Azure ingestion, privacy scrub, and viewer. The position requires treating privacy as a hard requirement throughout the development process. Key Responsibilities:• Design and deploy an OpenTelemetry capture path that includes endpoint environment variable setup, OTLP export, and a collector pipeline with receivers, processors, and exporters.• Implement the central ingestion and privacy‑scrub block in Azure, incorporating secret and PII detection, redaction, and tokenization before any data reaches a reviewer.• Develop and configure sanitized storage solutions and a read‑only trace viewer that maintains task, session, and trace correlation.• Write and maintain Bash scripts and supporting code in Python or Node.js to automate deployment, monitoring, and maintenance of the pipeline components.• Ensure cloud‑security best practices are applied, including RBAC, private networking, and audit logging for all Azure resources used. Required Skills & Qualifications:• Bachelor’s degree in computer science, engineering, or a related field, or equivalent practical experience, with at least five years of software, platform, or data engineering experience delivering production systems.• Minimum three years of hands‑on experience with Azure PaaS services such as Container Apps, AKS, API Management, ADX/Kusto, App Insights, ADLS Gen2, Key Vault, and Entra ID.• At least two years of direct experience building observability or telemetry pipelines using OpenTelemetry, including OTLP, collectors, spans/traces, and semantic conventions.• Proficiency in Python or Node.js combined with strong Bash scripting abilities for automation and integration tasks.• Demonstrated experience with data privacy, PII detection, and secret scrubbing techniques using regex, entropy analysis, or tokenization, and familiarity with cloud‑security fundamentals. More About the Opportunity:This role offers a chance to work with a global leader in AGI infrastructure, contributing to the creation of high‑quality training data and evaluation pipelines for frontier AI labs. The work directly supports leading large‑language‑model companies in accelerating model development while maintaining rigorous privacy standards. Equal Opportunity Employer:We hire based on skills and expertise. All qualified candidates are welcome regardless of background, experience, or prior employment history. Applications are reviewed solely on demonstrated technical ability and qualifications. Apply Now!