Lead Data Engineer
About the role
Lead Data Engineer - Remote position available for a temporary to permanent opportunity. Responsibilities include architecting, building, and optimizing distributed data pipelines using Apache Spark, designing and maintaining enterprise Lakehouse architecture with Delta Lake, developing automated ingestion frameworks, enabling machine-learning workflows, leading data quality and access control frameworks, implementing cost-optimization strategies, collaborating with various teams, and owning production reliability.
Responsibilities
- Architect, build, and optimize distributed data pipelines using Apache Spark in a high-volume, mission-critical environment.
- Design and maintain enterprise Lakehouse architecture with Delta Lake, ensuring ACID compliance, lineage, auditability, and data governance.
- Develop automated ingestion frameworks (batch, streaming, and event-driven) across multiple cloud services and integration points.
- Enable machine-learning workflows by preparing feature-ready datasets and establishing reproducible ML deployment patterns.
- Lead platform-wide data quality, access control, and cataloging frameworks.
- Implement advanced cost-optimization, cluster tuning, and performance engineering strategies.
- Collaborate with Finance, BI, Operations, and ML teams to translate complex business needs into scalable data solutions.
- Own production reliability, troubleshooting, and root-cause analysis for data and ML pipelines.
Requirements
- US Citizen, GC Holders or Authorized to Work in the U.S.
- 7+ years of experience in advanced data engineering with distributed compute technologies.
- Expert-level Spark engineering (performance tuning, cluster configuration, partition strategies, optimization of large datasets).
- Hands-on experience with Lakehouse architectures including ACID transactions, schema evolution, and governance frameworks.
- Deep proficiency in Python and SQL for large-scale data transformation.
- Experience supporting machine-learning pipelines or model operationalization.
- Proven experience architecting cloud-native data platforms (Azure, AWS, or GCP).
- Strong background integrating diverse, complex data sources at enterprise scale.
- Demonstrated ability to own mission-critical production systems.
Skills
- Core Tools: Databricks (Spark, Delta Lake, MLflow, Notebooks), Python & SQL, Apache Spark (via Databricks), Delta Lake (for lakehouse architecture), Cloud Platforms (Azure, AWS, or GCP), Cloud Storage (ADLS, S3, GCS), Data Integration (Kafka or Event Hubs), Auto Loader (Databricks file ingestion), REST APIs, AI/ML (MLflow, model tracking/deployment), Hugging Face Transformers, LangChain / LlamaIndex (LLM integration), LLMs (Anthropic Claude, Meta LLaMA, Google Gemini), DevOps (Git, GitHub, GitLab, Azure Repos), Databricks Repos, CI/CD (GitHub Actions, Azure DevOps), Security & Governance (Unity Catalog, RBAC)
- Experience with distributed streaming frameworks (Kafka, Event Hubs, or similar).
- Experience building or supporting ML platforms, feature stores, or experiment-tracking systems.
- Background in data security, compliance controls, or audit-ready governance.
- Experience automating data operations with CI/CD and infrastructure-as-code.
Benefits
Comprehensive medical benefits
Competitive pay
401(k) retirement plan
…and much more!
About INSPYR Solutions
INSPYR Solutions is a national expert in delivering flexible technology and talent solutions. We strategically align industry and technical expertise with our clients' business objectives and cultural needs. Our solutions are tailored to each client and include a wide variety of professional services, project, and talent solutions. By always striving for excellence and focusing on the human aspect of our business, we work seamlessly with our talent and clients to match the right solutions to the right opportunities.
Equal Employment Opportunity
Inspyr Solutions provides Equal Employment Opportunities (EEO) to all employees and applicants for employment without regard to race, color, religion, sex, national origin, age, disability, or genetics. In addition to federal law requirements, Inspyr Solutions complies with applicable state and local laws governing nondiscrimination in employment in every location in which the company has facilities.