Data Engineer

What You Will Be Doing

Design, develop, and implement robust, scalable, and high-performance data pipelines and applications using Python, PySpark, and Big Data technologies.

Work autonomously to analyze requirements, propose technical solutions, and deliver high-quality code and data products, ensuring alignment with architectural standards and business objectives.

Utilize expertise in various Big Data platforms (e.g., Hadoop, Hive, Kafka, Spark) to process, transform, and manage large datasets efficiently.

Write complex SQL queries, stored procedures, and optimize database performance for large-scale data warehousing and analytics solutions.

Develop and enhance ETL (Extract, Transform, Load) processes, ensuring data quality, integrity, and timely delivery. Experience with various ETL tools and methodologies is a plus.

Proactively research, evaluate, and integrate new and emerging technologies, frameworks, and tools to improve development processes and solution capabilities.

Ensure adherence to coding standards, conduct thorough code reviews, and implement best practices for software development, data governance, and security.

Diagnose and resolve complex technical issues related to data pipelines, performance bottlenecks, and system integrations in a fast-paced environment.

Collaborate effectively with cross-functional teams including architects, data scientists, business analysts, and QA engineers. Provide technical guidance and mentorship to junior team members.

Identify opportunities to use AI tools to speed up development, code reviews, unit testing and deployment.

Required Skills & Experience

• Bachelor's degree.
• 10+ years of experience in Applications Development, Systems Analysis, or equivalent senior engineering roles.
• Deep expertise in Python, PySpark, Big Data technologies, and SQL, coupled with a strong ability to work independently and contribute significantly to complex data engineering initiatives.
• Proven track record in designing, developing, and optimizing scalable data solutions, with experience in ETL processes and a keen interest in leveraging the latest technologies.
• Extensive hands-on experience delivering enterprise-scale, database-driven platforms in a regulated environment.
• Expert-level proficiency in Python programming, including object-oriented design, data structures, algorithms, and extensive experience with various Python libraries.
• Strong understanding of Apache Spark architecture, Hadoop ecosystem, and experience with distributed computing concepts. Familiarity with big data storage formats (e.g., Parquet, ORC).
• Strongly prefer domain knowledge in Finance.

Benefits

• Medical Insurance - Four medical plans to choose from for you and your family
• Dental & Orthodontia Benefits
• Vision Benefits
• Health Savings Account (HSA)
• Health and Dependent Care Flexible Spending Accounts
• Voluntary Life Insurance, Long-Term & Short-Term Disability Insurance
• Hospital Indemnity Insurance
• 401(k) including match with pre and post-tax options
• Paid Sick Time Leave
• Legal and Identity Protection Plans
• Pre-tax Commuter Benefit
• 529 College Saver Plan