Principal Data Engineer (New York)

How You'll Make an Impact

- Lead the design and architecture of end-to-end data pipelines and solutions on modern cloud-based platforms, including Snowflake, Databricks, and AWS.

- Lead the design and architecture of end-to-end data pipelines and solutions on platforms including Java, Springboot, Docker/Kubernetes, Snowflake, Databricks, and AWS.

- Design and implement data solutions using PostgreSQL for relational data and MongoDB for NoSQL requirements, ensuring optimal performance and scalability.

- Architect and deploy containerized data applications using Docker, Kubernetes, and AWS EKS, incorporating GitHub Actions for automated deployments.

- Design and implement CI/CD pipelines using GitHub Actions, establish branching strategies, and ensure automated testing, code quality checks, and security scanning.

- Collaborate with cross-functional teams—including Data Scientists, Analytics teams, and business stakeholders—to translate requirements into scalable technical solutions.

- Mentor and guide data engineers by promoting technical excellence, establishing coding standards, and conducting architecture reviews.

- Drive data platform modernization initiatives and ensure data quality, reliability, and governance across all data systems.

- Design and implement AI-enhanced data pipelines that leverage LLMs and Agentic AI frameworks to automate data quality checks, anomaly detection, and intelligent data transformation workflows.

- Architect data infrastructure to support AI/ML workloads, including feature stores, vector databases, and real-time inference pipelines integrated with cloud-native services.

- Leverage established standards and best practices to integrate AI agents into data engineering workflows, including context management protocols (MCP) for seamless AI-to-data-platform communication.

You May Be a Good Fit If

• You have 8+ years of data engineering experience, including 3+ years in a lead role architecting large-scale data platforms.
• Expert-level proficiency in Java, Springboot for building cloud-native data processing solutions running on Docker/Kubernetes.
• Deep hands-on experience with Apache Airflow, Snowflake (data warehousing, modeling, optimization), and Databricks.
• Strong AWS expertise including S3, Lambda, Glue, EMR, Kinesis, EKS, RDS etc.
• Production database experience with PostgreSQL (design, optimization, replication) and MongoDB (document modeling, sharding, replica sets).
• You have proven CI/CD and GitOps experience using GitHub, GitHub Actions, and ArgoCD for automated deployments and multi-environment management.
• You are proficient with agile tools such as JIRA for sprint management and Confluence for technical documentation and knowledge sharing.
• You have excellent analytical, problem-solving, and communication skills, with the ability to explain complex concepts to non-technical stakeholders and drive initiatives in complex environments.
• You have working knowledge of AI/ML frameworks (LangChain, LlamaIndex, AutoGen, etc.) and understand how Agentic AI can enhance data engineering workflows through automated data validation, intelligent orchestration, and self-healing pipelines.
• You have practical understanding of AI integration patterns in data platforms, including prompt engineering, RAG architectures, and vector database implementations.
• You are familiar with Model Context Protocol (MCP) or similar frameworks for enabling AI agents to interact securely and efficiently with data sources, APIs, and tools.
• You have experience with AI-powered development tools such as GitHub Copilot and Amazon Q.

Why Choose Fitch

• A hybrid work environment with on-site presence required two days per week.
• A culture of learning & mobility with access to dedicated training, leadership development, and mentorship programs.
• Investing in your future with retirement planning and tuition reimbursement programs.
• Promoting health & wellbeing with comprehensive healthcare offerings.
• Supportive parenting policies including a generous global parental leave plan.
• An inclusive work environment where all voices are valued, supported by Employee Resource Groups.
• A dedication to giving back with paid volunteer days, matched donation programs, and ample opportunities to volunteer in your community.