Lead Data Engineer
Saur Energy International · Location, WV · Yesterday
Full-time
Key Responsibilities
- Design and develop scalable data pipelines and Lakehouse solutions on Databricks.
- Tune Databricks workloads for performance and cost, including cluster sizing, query optimization, and Delta Lake table design.
- Establish and enforce best practices for partitioning, clustering, and workload isolation.
- Track performance trends, identify high-cost queries, and partner with source teams and end users to resolve long-running loads.
- Build monitoring and self-healing automation using Databricks-native AI and agentic capabilities.
- Drive CI/CD workflows for Databricks assets, setting DevOps best practices for deployment and release management.
- Lead design reviews and mentor Data Engineers on Databricks best practices and AI-native features.
- Owning Databricks vendor coordination — case management, escalations, and release adoption strategy.
Required Qualifications
- Bachelor's or Master's degree in Computer Science, Information Technology, or equivalent relevant experience.
- 6+ years of data engineering experience with 2+ years hands-on Databricks in enterprise settings.
- Deep understanding of Databricks Lakehouse architecture, Delta Lake, Unity Catalog, and Workflow orchestration.
- Proven ability to tune Spark workloads for cost and performance at production scale.
- Advanced Python (PySpark) and SQL skills.
- Working knowledge of CI/CD practices and DevOps principles applied to data workloads.
- Experience with observability tooling for Databricks.
Preferred Qualifications
- Experience with Databricks-native AI capabilities and agentic frameworks.
- Familiarity with Databricks Serverless Compute and DBSQL performance tuning.
- Databricks Certified Professional.
- Exposure to Infrastructure-as-Code.
Competencies
- Performance-engineering mindset — measures, tunes, and re-measures.
- Curiosity for AI-native operations and continuous automation.
- Strong sense of platform ownership — quality, cost, and reliability.
- Effective communication with engineering peers, vendors, and business stakeholders.
- Influence outcomes across source teams, vendors, and business stakeholders without direct authority.