Python Developer

Job Description

Core Python & Data Engineering Expert-level programming skills in Python and PySpark, specifically for developing Spark applications. Experience with PySpark for data processing. Familiarity with data manipulation libraries (Pandas, NumPy). Scripting for automation and data orchestration. Complex query writing, including subqueries, window functions, and performance tuning. Strong proficiency in Spark Core, Spark SQL, Spark Streaming.

Beneficial: Experience with Spark GraphX. Experience with Spark performance optimization techniques (e.g., caching, partitioning, shuffle optimizations, memory management).

Required Minimum Education

Bachelor’s Degree

Must Have Skills/Attributes

• Spark
• Spark SQL
• SQL
• Experience with Spark GraphX
• Experience with Spark performance optimization techniques (e.g., caching, partitioning, shuffle optimizations, memory management)
• Big Data Ecosystem
• In-depth understanding of HDFS architecture, data storage, and fault tolerance mechanisms
• Experience with HDFS commands and administration
• Solid understanding of YARN resource management and job scheduling
• Fundamental understanding of the MapReduce programming paradigm (even if primary development is in Spark/Flink)
• Knowledge of Zookeeper for distributed coordination services
• Experience with Big Data Technologies (Apache Spark, Hadoop, Kafka) and data warehousing
• Database & Data Modeling
• Experience with HBase (for real-time access to large datasets within Hadoop)
• Experience with NoSQL databases such as Cassandra, MongoDB, or similar
• Familiarity with RDBMS concepts and SQL for data integration
• Understanding of dimensional modeling, fact and dimension tables, and star/snowflake schemas

Responsibilities

Lead hands-on implementation and architecture of Python frameworks for scalable, resilient, and performant applications. Develop solutions adhering to architectural standards, utilizing in-depth technical and business domain knowledge. Design and implement agentic flows and build autonomous AI agents capable of reasoning, planning, and tool utilization. Enforce industry-standard SDLC best practices and contribute to code quality through comprehensive code reviews. Collaborate effectively with cross-functional teams (Risk, Quants, FO, DevOps, Production Support) to ensure seamless project delivery. Drive development for Stress Testing and Regulatory Risk projects, specifically within the Market Risk domain. Manage project technical aspects, including planning, governance, and taking ownership of initiatives from conception to completion. Deliver high-quality solutions independently within a globally matrixed environment and under tight deadlines.

Benefits

For information and details on employment benefits offered with this position, please visit here. Should you have any questions/concerns, please contact our HR Department via our secure website.

California Pay Equity

For information and details on pay equity laws in California, please visit the State of California Department of Industrial Relations' website here.

About the Role

C2C is not available

Qualifications

Must Have Skills/Attributes: Spark, Spark SQL, SQL Experience Desired: Core Python & Data Engineering (6-10 yrs); Big Data Ecosystem (6-10 yrs); Database & Data Modeling (6-10 yrs)

Skills

Core Python & Data Engineering Expert-level programming skills in Python and PySpark, specifically for developing Spark applications. Experience with PySpark for data processing. Familiarity with data manipulation libraries (Pandas, NumPy). Scripting for automation and data orchestration. Complex query writing, including subqueries, window functions, and performance tuning. Strong proficiency in Spark Core, Spark SQL, Spark Streaming.

Beneficial: Experience with Spark GraphX. Experience with Spark performance optimization techniques (e.g., caching, partitioning, shuffle optimizations, memory management).

Requirements

• Core Python & Data Engineering Expert-level programming skills in Python and PySpark, specifically for developing Spark applications.
• Experience with PySpark for data processing.
• Familiarity with data manipulation libraries (Pandas, NumPy).
• Scripting for automation and data orchestration.
• Complex query writing, including subqueries, window functions, and performance tuning.
• Strong proficiency in Spark Core, Spark SQL, Spark Streaming.
• Beneficial: Experience with Spark GraphX.
• Experience with Spark performance optimization techniques (e.g., caching, partitioning, shuffle optimizations, memory management).
• Big Data Ecosystem In-depth understanding of HDFS architecture, data storage, and fault tolerance mechanisms.
• Experience with HDFS commands and administration.
• Solid understanding of YARN resource management and job scheduling.
• Fundamental understanding of the MapReduce programming paradigm (even if primary development is in Spark/Flink).
• Knowledge of Zookeeper for distributed coordination services.
• Experience with Big Data Technologies (Apache Spark, Hadoop, Kafka) and data warehousing.
• Database & Data Modeling Experience with HBase (for real-time access to large datasets within Hadoop).
• Experience with NoSQL databases such as Cassandra, MongoDB, or similar.
• Familiarity with RDBMS concepts and SQL for data integration.
• Understanding of dimensional modeling, fact and dimension tables, and star/snowflake schemas.