Applied Data Scientist, Finance AI Evaluation & Datasets

About the role

Innodata is seeking an Applied Data Scientist, Financial AI Evaluation & Datasets to join our team. This role focuses on the design, measurement quality, and domain validity of datasets used to train, fine-tune, evaluate, and monitor financial-domain LLMs, vision-language models, multimodal document models, and AI agents.

Responsibilities

• Translate customer goals into concrete dataset specifications, taxonomies, rubrics, and acceptance criteria.

• Design training and evaluation datasets across the financial AI surface: financial QA, filings and earnings analysis, credit and underwriting, fraud/AML investigation, and compliance, among other financial workflows.

• Foreground unstructured and multimodal financial data in dataset design — PDFs, scanned statements, tables, charts, and call transcripts — used by analysts, advisors, compliance reviewers, and operations teams.

• Evaluate agentic and workflow-integrated financial AI systems: tool use, retrieval, transaction boundaries, escalation behavior, and controls that prevent unsafe or unauthorized actions.

• Develop evaluation methodology that goes beyond surface accuracy — numerical consistency, hallucination rates on high-risk claims, refusal and escalation appropriateness, robustness under ambiguity, and fairness across protected or sensitive customer segments.

• Build the statistical and ML tooling that makes large financial datasets trustworthy: stratified sampling across products, markets, and modalities; bias analysis; leakage detection; and distribution shift checks, among other reliability checks.

• Build evaluation and dataset-quality evidence to support financial-services model risk management: assumptions, limitations, validation results, and residual risks, packaged as reproducible evidence.

• Partner with the AI/ML Research Engineer to instrument datasets into training, evaluation, and monitoring pipelines — rubric-grounded LLM-as-judge prompts, regression suites, and continuous monitoring.

• Own data quality end-to-end, from intake through delivery: PII handling, provenance tracking, versioning, and modality-specific QA checks.

• Reason about financial workflow context: where AI outputs enter analyst, advisor, compliance, risk, or customer-facing workflows; what evidence a reviewer needs to trust them; and when uncertainty must be surfaced.

• Support the Technical Solutions Architect during customer discovery and proposals: scoping dataset programs, sizing annotation effort, and explaining methodology to client stakeholders.

• Stay current on the financial AI landscape: regulatory developments, benchmark releases, and emerging evaluation methodology for finance-domain models.

• Contribute to Innodata internal IP: reusable taxonomies, evaluation rubrics, golden datasets, and methodology templates.

Requirements

• 5+ years of data science experience, with at least 2+ years in financial services, fintech, banking, or a comparable regulated data environment.

• Real working knowledge of financial data and workflows: financial statements, SEC filings, transaction data, and other common financial-services document types.

• Hands-on experience with unstructured and multimodal financial data — some combination of PDFs, scanned documents, spreadsheets, charts, or call transcripts.

• Familiarity with financial standards or protocols such as XBRL, ISO 20022, or GAAP/IFRS reporting concepts, etc. is strongly preferred.

• Hands-on experience designing datasets for ML — not just consuming them.

• You have written annotation guidelines, sized cohorts, set quality thresholds, and shipped data that downstream teams could actually train, evaluate, or monitor on.

• Familiarity with LLM-based and multimodal financial AI workflows: prompt design, rubric-based evaluation, RAG, LLM-as-judge methods, and the limitations of automated evaluation in high-stakes contexts.

• Strong Python and SQL; comfort with pandas, scikit-learn, or equivalent; working familiarity with Hugging Face, PyTorch, or model APIs.

• Statistical literacy: sampling design, inter-annotator agreement metrics (e.g., Cohen's kappa), confidence intervals, and the ability to push back when a number is being over-interpreted.

• Solid grasp of financial services privacy, compliance, and governance: PII handling, GLBA or equivalent privacy regimes, MNPI sensitivity, and documentation fit for regulated AI programs.

• Excellent collaboration skills — upstream with a Technical Solutions Architect, sideways with research scientists and engineers, and downstream with SME annotators and quality teams.

• A bias toward financial workflow realism.

Qualifications

• Degree in a relevant field — statistics, data science, economics, finance, or a related quantitative field, or equivalent demonstrated experience.

• Formal finance credentials aren't required, but CFA, FRM, or MBA backgrounds, etc. are especially encouraged.

• Experience designing evaluations for LLMs, VLMs, or multimodal models in financial reasoning, filings analysis, or fraud/AML contexts.

• Experience with document AI, OCR/post-OCR quality, or table and chart extraction for complex financial documents.

• Familiarity with agentic evaluation, AI observability, experiment tracking, or tools such as Weights & Biases or LangFuse.

• Familiarity with model risk management frameworks, validation documentation, fairness/bias auditing, or consumer protection analysis.

• Experience with multilingual or cross-border financial data, or published/open-source work in financial AI or model governance.

Benefits

The expected salary range for this position is $150,000 – $175,000 USD per year, based on experience, skills, and qualifications.

Pay

The expected salary range for this position is $150,000 – $175,000 USD per year, based on experience, skills, and qualifications.

Schedule

Full-time, remote position.