s/w engg

Job Summary

Responsibilities

• Design robust data processing pipelines using Python and PySpark in Databricks to transform complex data into reliable curated datasets that enable advanced analytics and reporting across the organization.
• Develop optimized Databricks SQL queries to support performant dashboards and analytical workloads ensuring data consumers can access trusted information with minimal latency and high reliability.
• Configure and manage Databricks Workflows to orchestrate end to end data jobs ensuring timely execution effective dependency handling and consistent delivery of data assets to downstream applications.
• Implement reusable modular code components and libraries in Python that standardize data transformations and business logic leading to faster development cycles and improved maintainability.
• Collaborate closely with data engineers analysts and product teams in a hybrid work model to gather requirements validate solutions and align data products with strategic business objectives.
• Optimize PySpark jobs for performance and cost efficiency by tuning configurations managing partitioning strategies and leveraging best practices for large scale distributed processing.
• Ensure data quality and reliability by implementing validation checks error handling mechanisms and logging standards that reduce data issues and support faster troubleshooting.
• Conduct thorough unit testing integration testing and performance benchmarking for Databricks workflows to ensure stable releases and minimize disruptions to data consumers.
• Document datasets workflows and code logic in clear technical artifacts so that other team members can understand reuse and extend solutions effectively over time.
• Provide support for day shift operations by monitoring data workflows resolving production issues and proactively identifying opportunities to improve data reliability without requiring travel.
• Engage in continuous improvement by evaluating new Databricks features Python libraries and data engineering practices that can enhance scalability security and usability of data platforms.
• Collaborate with cloud and security teams to ensure that all Databricks development adheres to organizational policies governance standards and compliance requirements benefiting stakeholders and society through responsible data use.
• Participate in code reviews and knowledge sharing sessions in the hybrid work environment to promote high quality development practices and foster a culture of continuous learning and innovation.

Qualifications

• Demonstrate strong proficiency in Python programming with hands on experience in writing efficient modular and testable code for data engineering and analytics use cases.
• Show practical expertise in authoring and optimizing Databricks SQL queries for complex joins aggregations and analytical functions that support business reporting and data exploration.
• Exhibit solid experience in building scheduling and managing Databricks Workflows including job orchestration parameterization and monitoring to support dependable data operations.
• Apply deep understanding of PySpark fundamentals including RDDs DataFrames and Spark SQL along with performance tuning techniques suitable for large scale distributed data processing.
• Bring relevant experience of four to seven years in data engineering or analytics focused development roles working on hybrid work models and collaborating with cross functional teams.
• Display strong communication and problem solving skills that enable effective collaboration with technical and nontechnical partners while focusing on delivering impactful data solutions.