AI Systems Engineer - Data & State Management - Senior

About the role

We are seeking an AI Systems Engineer to own the stateful backbone of EY’s AI-native platform. This role ensures that agentic AI workloads have durable, performant, and consistent access to data across various environments.

Responsibilities

• Own the memory and data stores: relational and durable state (PostgreSQL, DBOS durable workflows), caching (Redis/Valkey), vector stores (Qdrant/Milvus/PGVector), knowledge graphs (Neo4j), and object/block storage (MinIO, OpenEBS Mayastor).
• Own event streaming and async messaging: Apache Kafka (Strimzi), NATS JetStream (agent-to-agent), Debezium (change data capture), Apache Flink (stream processing), and Apicurio/CloudEvents (schema and event contracts).
• Build and operate streaming and CDC pipelines that move data reliably between stores and services, with schema governance and evolution that prevents breaking changes across producers and consumers.
• Make state multi-tenant and portable, ensuring isolation, performance, and consistent semantics whether running on managed cloud services or self-hosted OSS in an air-gapped environment.
• Provide the data and lineage substrate that downstream governance, observability, and AI knowledge capabilities depend on, as well as integrate with lineage tooling.

Requirements

• Deep expertise operating production databases and data stores at scale, including relational, key-value, vector, graph, and object storage.
• Strong command of streaming and event-driven architectures (Kafka, NATS, CDC, stream processing) and the consistency tradeoffs they involve.
• A durability-first mindset: thinking in terms of consistency, recoverability, blast radius, and data correctness under failure.
• Ability to operate stateful systems consistently across managed cloud and self-hosted OSS in cloud, on-prem, edge, and air-gapped environments.
• Strong grasp of schema governance and evolution, preventing breaking changes across producers and consumers.
• Strong communicator able to guide consuming teams toward the right storage and streaming patterns.
• Orientation toward reliability and toil reduction through automation and infrastructure-as-code for data systems.

Skills and Attributes

• Hands-on expertise with relational databases (PostgreSQL) and caching (Redis/Valkey), including HA, replication, and backup/recovery.
• Exposure to AI/ML data patterns — embeddings, retrieval, feature/state stores for agentic workloads.
• Production experience with vector and/or graph databases (Qdrant, Milvus, PGVector, Neo4j) in AI/ML contexts.
• Experience with stream processing (Apache Flink) and event/schema governance (CloudEvents, schema registry).
• Experience running stateful systems on Kubernetes (operators, persistent volumes, object/block storage such as MinIO/OpenEBS).
• Experience defining clean ownership boundaries and data/schema contracts with platform, trust, runtime, and delivery teams.
• Experience with durable workflow engines (DBOS, Temporal, or equivalents).
• Experience with data lineage and metadata tooling (OpenLineage, Marquez, or equivalents).
• Experience operating data systems under compliance, security, or regulatory constraints, including data-at-rest and in-transit protection.
• Experience with multi-tenant data isolation and per-tenant performance management.
• Experience with cross-environment replication and DR strategies tiered by RPO/RTO (e.g., MirrorMaker, CloudNativePG PITR).
• Exposure to regulated delivery environments (financial services, tax, healthcare, risk).

Pay

The base salary range for this job in the US is $106,900 to $176,500. The base salary range for New York City Metro Area, Washington State and California (excluding Sacramento) is $128,400 to $200,600. Individual salaries within those ranges are determined through a wide variety of factors including but not limited to education, experience, knowledge, skills and geography.

Schedule

We offer a flexible vacation policy where you'll decide how much vacation time you need based on your own personal circumstances. You'll also be granted time off for designated EY Paid Holidays, Winter/Summer breaks, Personal/Family Care, and other leaves of absence when needed to support your physical, financial, and emotional well-being.