Senior Data Engineer – Databricks & AWS

Job Title: Senior Data Engineer Databricks & AWS Location: San Jose, California, USA Work Mode: Strictly Onsite Employment Type: Contract Experience Required: 12+ Years Job Summary We are seeking an experienced and hands-on Senior Data Engineer with strong expertise in Databricks and AWS to design, develop, and scale an enterprise-grade data platform. The ideal candidate will have extensive experience building reliable, scalable, governed, and reusable data products that support enterprise analytics, machine learning, Tableau reporting, and AI-driven self-service analytics. The candidate will be responsible for developing modern lakehouse solutions, implementing robust data engineering pipelines, establishing enterprise data models, and ensuring data quality, governance, security, and performance across the data platform. This role requires strong technical expertise, architectural thinking, independent problem-solving skills, and the ability to collaborate effectively with business and technical stakeholders. Key Responsibilities Design, develop, and maintain scalable data pipelines and enterprise data solutions using Databricks, Apache Spark, PySpark, Python, SQL, and AWS.Build and optimize data processing workflows using Delta Lake and the Medallion Architecture (Bronze, Silver, and Gold layers).Develop reliable, reusable, and production-ready data products to support analytics, reporting, machine learning, and AI applications.Design and implement enterprise data models, including fact tables, dimension tables, and semantic layers.Implement data governance, access controls, data lineage, and metadata management using Unity Catalog.Develop and optimize complex SQL queries, Spark jobs, and large-scale data transformations for performance and cost efficiency.Establish and maintain CI/CD pipelines, Git-based version control, automated testing, and DataOps best practices.Leverage AWS cloud services to build, integrate, and operate scalable data engineering solutions.Analyze source systems, gather business and technical requirements, and translate them into production-ready data solutions.Ensure data quality, consistency, reliability, security, and compliance across the data lifecycle.Collaborate with data architects, analysts, data scientists, business stakeholders, and application teams to deliver enterprise data solutions.Troubleshoot production issues, optimize existing pipelines, and support continuous improvement of the data platform. Required Skills and Qualifications 12+ years of hands-on experience in enterprise data engineering and large-scale data platform development.Strong hands-on experience with Databricks in production environments.Advanced proficiency in SQL, Python, Apache Spark, and PySpark.Practical experience with Delta Lake, lakehouse architecture, and Medallion Architecture.Strong understanding of enterprise data modeling, including facts, dimensions, data marts, and semantic layers.Experience with Unity Catalog, data governance, metadata management, data lineage, and access controls.Hands-on experience with Git, CI/CD pipelines, automated deployments, and DataOps practices.Strong experience with AWS cloud services and cloud-based data engineering architectures.Proven ability to independently manage requirements from source system analysis and data modeling through development, deployment, and production support.Strong analytical, troubleshooting, communication, and stakeholder management skills. Preferred / Good-to-Have Skills Experience with Databricks Genie, conversational AI, or self-service analytics solutions.Knowledge of AI-ready semantic layers, contextual data modeling, and data products for generative AI applications.Exposure to machine learning, feature engineering, or MLOps.Experience working with manufacturing, supply chain, ERP, CRM, PLM, or MES data.Experience developing and optimizing Tableau reporting datasets and analytics solutions.Experience with cloud lakehouse modernization, legacy data platform migration, and enterprise data transformation initiatives.