Hadoop Hive Python Developer

Key Responsibilities

• Design, develop, and optimize PySpark-based ETL pipelines running on on-prem Hadoop clusters and cloud environments.
• Build high-volume ingestion frameworks using Kafka for real-time and near-real-time trading and market data.
• Develop, tune, and manage Hadoop ecosystem components—HDFS, YARN, MapReduce, Tez, Oozie/Airflow.
• Build high-performance, optimized Hive data models for regulatory reporting, trade lifecycle, and market risk processing.
• Databricks Lakehouse & Delta Framework Architect and implement Bronze/Silver/Gold layer modeling patterns within the Databricks Lakehouse.
• Apply Delta Lake best practices including: optimized file management, Z-Ordering, Delta Change Data Feed (CDF), schema evolution & enforcement, ACID transaction handling.
• Build reusable frameworks for ingestion, cleansing, transformation, and consumption of data across Lakehouse layers.
• Enable governance, lineage, and auditability using Unity Catalog or equivalent cataloging tools.
• Collaborate closely with quants, product owners, architects, risk tech, and business users.
• Participate in agile ceremonies — sprint planning, refinement, design reviews.
• Mentor junior engineers and contribute to building strong engineering practices across tech teams.

Required Skills & Experience

• 9-14 years of hands-on experience in Big Data engineering.
• Expert skills in: PySpark — dataframe optimizations, partitioning, broadcast strategies, distributed computing; Kafka — producer/consumer design, schema registry, streaming ETLs; Hadoop ecosystem — HDFS, YARN, MapReduce/Tez, Oozie/Airflow; Hive — advanced query tuning, TEZ optimization, partition/bucket management.
• Extensive hands-on experience with Databricks Lakehouse, including: Bronze/Silver/Gold layer modeling, Delta Lake optimizations, data quality frameworks on Lakehouse, structured & unstructured data handling.
• Experience in Global Markets, Risk, Treasury, Trade Surveillance, or Regulatory Reporting.
• Strong SQL knowledge with experience working on massive datasets (TB/PB scale).
• Experience with CI/CD practices — Git, Jenkins, Bitbucket, build pipelines.

TCS Employee Benefits Summary

• Discretionary Annual Incentive.
• Comprehensive Medical Coverage: Medical & Health, Dental & Vision, Disability Planning & Insurance, Pet Insurance Plans.
• Family Support: Maternal & Parental Leaves.
• Insurance Options: Auto & Home Insurance, Identity Theft Protection.
• Convenience & Professional Growth: Commuter Benefits & Certification & Training Reimbursement.
• Time Off: Vacation, Time Off, Sick Leave & Holidays.
• Legal & Financial Assistance: Legal Assistance, 401K Plan, Performance Bonus, College Fund, Student Loan Refinancing.

Salary Range

$110,000-$125,000 a year