Senior Python Data Engineer (PySpark & Big Data)

Job Description

Lead hands-on implementation and architecture of Python frameworks for scalable, resilient, and performant applications.

Develop solutions adhering to architectural standards, utilizing in-depth technical and business domain knowledge.

Design and implement agentic flows and build autonomous AI agents capable of reasoning, planning, and tool utilization.

Enforce industry-standard SDLC best practices and contribute to code quality through comprehensive code reviews.

Collaborate effectively with cross-functional teams (Risk, Quants, FO, DevOps, Production Support) to ensure seamless project delivery.

Drive development for Stress Testing and Regulatory Risk projects, specifically within the Market Risk domain.

Manage project technical aspects, including planning, governance, and taking ownership of initiatives from conception to completion.

Deliver high-quality solutions independently within a globally matrixed environment and under tight deadlines.

Must Have Skills/Attributes

• Banking/Financial
• Big Data
• Flink
• Hadoop
• MapReduce
• PySpark
• Scala
• Spark
• SQL
• Zookeeper

Preferred Qualifications/Skills/Experience

• Spark GraphX

Preferred Education

Bachelor’s Degree

Required Qualifications/Skills/Experience

• Expert-level programming skills in Python and PySpark, specifically for developing Spark applications.
• Familiarity with data manipulation libraries (Pandas, NumPy).
• Scripting for automation and data orchestration.
• Complex query writing, including subqueries, window functions, and performance tuning.
• Strong proficiency in Spark Core, Spark SQL, Spark Streaming.
• Experience with Spark performance optimization techniques (e.g., caching, partitioning, shuffle optimizations, memory management).
• In-depth understanding of HDFS architecture, data storage, and fault tolerance mechanisms.
• Experience with HDFS commands and administration.
• Solid understanding of YARN resource management and job scheduling.
• Fundamental understanding of the MapReduce programming paradigm (even if primary development is in Spark/Flink).
• Knowledge of Zookeeper for distributed coordination services.
• Experience with Big Data Technologies (Apache Spark, Hadoop, Kafka) and data warehousing.
• Experience with HBase (for real-time access to large datasets within Hadoop).
• Experience with NoSQL databases such as Cassandra, MongoDB, or similar.
• Familiarity with RDBMS concepts and SQL for data integration.
• Understanding of dimensional modeling, fact and dimension tables, and star/snowflake schemas.

Benefits

For information and details on employment benefits offered with this position, please visit here.

If you need assistance in completing this application, or during any phase of the application, interview, hiring, or employment process, whether due to a disability or otherwise, please contact our HR Department via our secure website.

Rose International is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, age, sex, sexual orientation, gender (expression or identity), national origin, arrest and conviction records, disability, veteran status or any other characteristic protected by law.

Positions located in San Francisco and Los Angeles, California will be administered in accordance with their respective Fair Chance Ordinances.

If you need assistance in completing this application, or during any phase of the application, interview, hiring, or employment process, whether due to a disability or otherwise, please contact our HR Department.

Rose International has an official agreement (ID #132522), effective June 30, 2008, with the U.S. Department of Homeland Security, U.S. Citizenship and Immigration Services, Employment Verification Program (E-Verify).