Legal Domain Expert (Remote | Freelance)

About the Project

Deccan AI is building a Deep Research Agent (DRA) Benchmark for the Legal domain as part of its GDPVal evaluation study. The benchmark evaluates how well frontier AI agents (OpenAI o3-deep-research, Google Gemini Deep Research, Claude) perform on realistic, high-difficulty professional legal tasks.

Role Summary

You will serve as the lead legal authority for the DRA Benchmark programme. You will own four inter-related workstreams: finalising the legal taxonomy and sub-domain framework; designing and validating the AI-driven SME interviewer used to screen and onboard task-writing legal experts; calibrating the task complexity and scoring framework; and conducting QC reviews of submitted prompt packages to ensure they meet the benchmark's analytical rigour, legal reasoning, and verifiability standards.

Key Responsibilities

• Taxonomy Finalisation

  Review and validate the selected legal domains and formally document the rationale for inclusion and exclusion decisions.

  Define precise boundaries between overlapping legal domains, including:

  • Commercial Litigation (e.g., Litigation Risk Assessment and Strategy) vs. Corporate Law (e.g., Board Advisory and Compliance)
  • Regulatory Compliance (e.g., Industry-Specific Regulatory Frameworks (sectoral licensing, regulated industries, permitting regimes, etc.)) vs. Consumer Protection (e.g., Unfair and Deceptive Trade Practices)
  • Employment & Labor Law (e.g., Workplace Safety and Occupational Safety Regulations) vs. Regulatory Compliance (e.g., Administrative Hearings and Agency Proceedings)
  • Legal Research & Writing (e.g., Statutory and Regulatory Research) vs. Policy & Legislative Analysis (e.g., Regulatory Impact Analysis)
  • Contract Review & Management (e.g., IP Licensing Agreements overlap) vs. Intellectual Property (e.g., IP Licensing Agreements, IP Litigation and Settlement)

  Build and maintain a cross-domain persona capability matrix indicating which legal personas can author or review tasks across domains. Example personas:

  • Commercial Litigation Counsel
  • Corporate Counsel
  • Contracts Manager
  • Compliance Officer
  • IP Attorney
  • Employment Law Specialist
  • Environmental Counsel
  • Policy Analyst

  Validate and correct domain tags on all reference tasks and ensure consistent labeling across benchmark documentation.

• AI Interviewer Design for SME Onboarding

  Design the AI-driven screening interview used to evaluate incoming legal SMEs before onboarding.

  Define competency dimensions the interview must assess:

  • Legal technical depth
  • Statutory and regulatory interpretation
  • Case-law, precedent, and legal authority analysis
  • Ability to identify likely AI reasoning failures
  • Ability to write objective, binary-checkable verifiers
  • Legal drafting and analytical reasoning skills

  Author or review the screening question bank.

  Define pass/fail thresholds for each legal sub-domain and persona level.

  Determine when borderline candidates require secondary human review.

  Test and calibrate the AI interviewer against expert legal judgment before deployment.

• Task Complexity Framework Calibration

  Validate the Numerical Rigor Level (L0-L5) framework and identify misclassifications.

  Finalize legal prompt-type definitions and cognitive trap frameworks.

  Define complexity indicators such as:

  • Multi-jurisdictional analysis
  • Conflicting precedents
  • Regulatory ambiguity
  • Statutory interpretation challenges
  • Contractual conflict resolution
  • Litigation strategy trade-offs
  • Policy impact analysis

  Ensure complexity levels accurately reflect professional legal practice rather than simple legal knowledge retrieval.

• Quality Control (QC) of Submitted Prompt Packages

  Review submitted prompt packages against the four-component standard:

  • Prompt
  • Data Files
  • Solution Logic
  • Sanity Check

  Verify solution logic step-by-step and independently validate legal conclusions.

  Confirm legal authorities, statutes, regulations, precedents, and contractual provisions cited in solutions.

  Validate prompt-specific verifiers to ensure they are:

  • Binary-checkable
  • Legally grounded
  • Derived from the expected solution
  • Non-redundant with rubric criteria

  Maintain a QC log of recurring legal reasoning and task-design issues.

  Escalate benchmark calibration issues where tasks are systematically too easy or too difficult for evaluated models.

Required Qualifications

• Professional legal qualification required (e.g., J.D., LL.B., B.L., or equivalent law degree recognized in the candidate's jurisdiction).
• Minimum 10 years of post-qualification legal experience in law firms, corporate legal departments, regulatory bodies, consulting firms, policy organizations, or litigation practice.
• Experience working with, supervising, reviewing, or evaluating legal work products across multiple jurisdictions is strongly preferred.
• Candidates should be comfortable assessing legal reasoning, legal research, and legal work products originating from both United States and Indian legal contexts.
• Demonstrable expertise across at least three of the benchmark domains:
• Commercial Litigation
• Corporate Law
• Contract Review & Management
• Regulatory Compliance
• Intellectual Property
• Consumer Protection
• Employment & Labor Law
• Environmental Law
• Legal Research & Writing
• Policy & Legislative Analysis

Preferred Qualifications

• Professional governance, corporate secretarial, compliance, or corporate governance qualifications/experience preferred.
• Prior experience in AI evaluation, benchmark design, RLHF annotation, legal-tech products, or legal knowledge systems.
• Experience designing interviews, assessments, or competency frameworks for legal professionals.
• Cross-sector experience in litigation, corporate advisory, compliance, technology law, public policy, or regulatory affairs.