Senior Data Engineer

Description

At CDW, we make it happen, together. Trust, connection, and commitment are at the heart of how we work together to deliver for our customers. It’s why we’re coworkers, not just employees. Coworkers who genuinely believe in supporting our customers and one another. We collectively forge our path forward with a level of commitment that speaks to who we are and where we’re headed. We’re proud to share our story and Make Amazing Happen at CDW.

About the role

The Senior Data Engineer designs, builds, and maintains scalable data pipelines and Lakehouse solutions on the Databricks platform to support enterprise data and AI initiatives. This role partners closely with architects, analytics, and data science teams to deliver high-quality, reliable data products. The engineer operates with significant autonomy and contributes deep technical expertise across the full data lifecycle.

Responsibilities

• Build and maintain scalable data pipelines, ETL and ELT processes, and data models within the Databricks platform.
• Design, develop, and deploy data and AI solutions using Databricks, Spark, Delta Lake, and related technologies.
• Develop batch and streaming pipelines using tools such as Databricks Workflows and Azure Data Factory.
• Design logical data flow diagrams and normalized schemas, implementing Lakehouse patterns such as the Medallion Architecture (Bronze, Silver, Gold layers).
• Ensure data quality, integrity, security, and governance throughout the data lifecycle, including use of Unity Catalog.
• Optimize Spark jobs and data transformations through effective partitioning, caching, and join strategies.
• Monitor pipeline execution, identify failures, and troubleshoot complex data processing issues.
• Collaborate with data architects, analysts, data scientists, and business stakeholders to understand requirements and deliver solutions.
• Support documentation of data processes, standards, and data flows.

Requirements

• 5 Years of experience designing, developing, and deploying data solutions on the Databricks platform.
• Proficiency in Python, including PySpark, and SQL.
• Hands-on experience with Spark, Delta Lake, and Lakehouse architectures.
• Experience implementing data quality, governance, and security practices across data pipelines.
• Familiarity with machine learning concepts, tools, and libraries such as TensorFlow, PyTorch, Scikit-learn, and MLflow (plus).
• Experience configuring and integrating external AI models and working with AI governance and monitoring tools (plus).
• Experience with asynchronous programming patterns in Python for building scalable data or AI workloads (plus).
• Strong problem-solving, collaboration, and communication skills.

Qualifications

• Master's degree in Computer Science, Engineering, Statistics, or a related field (plus).
• Experience with cloud platforms such as Azure (plus).
• Experience with distributed systems and microservices architecture (plus).

Skills

• Python programming.
• SQL database management.
• Databricks platform expertise.
• Spark and Delta Lake technologies.
• Lakehouse architecture implementation.
• Data quality and governance practices.
• Machine learning concepts and tools.
• AI model integration and governance.
• Asynchronous programming in Python.

Benefits

• CDW is committed to fostering an equitable, transparent, and respectful hiring process for all applicants.
• We value adopting AI as a partner, openness to experimentation, and a shared interest in learning together on AI.
• CDW is an equal opportunity employer.

Pay

Competitive compensation package.

Schedule

Full-time position.