Lead Data Engineer

Job Description

The Lead Data Engineer is a senior technical leader responsible for guiding the design, development, and optimization of Visa’s large-scale data platforms and cloud-based analytics environments. This role provides architectural direction, leads complex engineering initiatives, and mentors teams while remaining deeply hands-on with modern data technologies.

• Lead the architecture and delivery of large-scale, high-performance data pipelines and processing frameworks across Hadoop and multi-cloud environments.
• Design scalable data models, lakehouse structures, and distributed data processing solutions that support analytics, machine learning, and real-time data needs.
• Provide technical leadership to Senior and Staff Data Engineers, conducting design reviews, guiding implementation decisions, and ensuring engineering excellence.
• Partner with cross-functional teams to translate business and product requirements into robust technical designs and data solutions.
• Develop and improve engineering best practices for data governance, quality, observability, testing, and cloud resource optimization.
• Drove adoption of cloud-native data technologies, automation frameworks, and reusable components that improve development velocity and system reliability.
• Lead complex data modernization efforts, including cloud migration, data lake/lakehouse consolidation, and performance optimization of critical pipelines.
• Evaluate new tools and technologies, influencing platform evolution within the scope of assigned domains or product areas.
• Collaborate with product, analytics, and platform teams to ensure alignment on data strategy and architectural roadmaps.
• Mentor engineers at all levels, providing technical coaching and fostering a culture of continuous improvement.

Qualifications

• Advanced expertise in building and optimizing large-scale distributed data systems using Hadoop, Spark, and modern lakehouse architectures.
• Strong programming proficiency in PySpark, Scala, and Python with experience implementing scalable, production-grade data applications.
• Deep experience designing and tuning RDBMS, NoSQL, and distributed SQL systems.
• Mastery of SQL and distributed query engines such as Presto, Trino, Hive, and SparkSQL.
• Proven experience architecting and operating data solutions on AWS, GCP, and Azure, including cloud data lakes, orchestration tools, and cost-effective storage/compute designs.
• Advanced proficiency in Databricks, including: Building and optimizing notebooks and production jobs Delta Lake design and optimization Cluster configuration and workspace administration CI/CD integration for data workloads Performance tuning for large distributed jobs.
• Demonstrated ability to lead technical initiatives, communicate architectural decisions, and influence engineering direction across multiple teams.
• Strong problem-solving skills with the ability to troubleshoot complex data and performance issues.