Senior Applied Scientist

Responsibilities

• Independently own end-to-end evaluations of foundation models, AI agents, and enterprise AI systems, from initial question and experiment design through analysis, reporting, and stakeholder review.
• Translate customer, product, and business needs into testable hypotheses, evaluation criteria, datasets, metrics, baselines, and acceptance thresholds.
• Design, implement, and maintain benchmarks and evaluation methods for areas such as but not limited to reasoning, coding, agentic workflows, RAG, NL2SQL, multimodal systems, multilingual performance, and responsible AI.
• Publish original research in top-tier peer-reviewed conferences and journals, and translate relevant evaluation advances into reusable methods, technical reports, or production capabilities for OCI.
• Write production-quality evaluation code; build reproducible pipelines, test suites, automated checks, and integrations with shared evaluation platforms.
• Evaluate model and system behavior across quality, cost, latency, reliability, safety, robustness, and domain fit rather than relying only on aggregate scores.
• Conduct statistical analysis, error analysis, ablations, and qualitative failure-mode investigations to explain model behavior and identify meaningful differences between systems.
• Develop and validate automated evaluators, including LLM-as-a-judge and VLM-as-a-judge methods; calibrate them against human judgments and quantify their reliability, bias, and limitations.
• Design human-evaluation and annotation workflows, including rubrics, gold datasets, sampling plans, quality controls, and vendor or Human-in-the-Loop validation.
• Assess dataset quality, provenance, representativeness, contamination risk, licensing constraints, privacy, and other factors that could invalidate an evaluation or limit use of its results.
• Produce concise, decision-ready reports that make methods, assumptions, limitations, tradeoffs, and recommendations explicit for technical and non-technical audiences.
• Partner with science, engineering, product, data, and operations teams to define requirements, resolve blockers, manage dependencies, and establish clear handoffs and ownership.
• Turn successful evaluation work into reusable protocols, documented workflows, and shared infrastructure that improve the speed and consistency of future evaluations.
• Stay current with research in machine learning, generative AI, agent evaluation, and measurement methodology; prototype promising approaches and contribute to science plans, papers, patents, or technical reports where appropriate.
• Review technical work, share expertise, and mentor junior scientists or engineers in experimental design, evaluation methodology, coding, and interpretation of results.
• Own delivery quality and timelines, communicate risks early, and maintain clear, auditable documentation of experimental configurations, data versions, results, and decisions.

Qualifications

• Range and benefit information provided in this posting are specific to the stated locations only US: Hiring Range in USD from: $114,600 - $234,600 per year. May be eligible for bonus, equity, and compensation deferral.
• Oracle maintains broad salary ranges for its roles in order to account for variations in knowledge, skills, experience, market conditions and locations, as well as reflect Oracle's differing products, industries and lines of business. Candidates are typically placed into the range based on the preceding factors as well as internal peer equity.
• Oracle US offers a comprehensive benefits package which includes the following: Medical, dental, and vision insurance, including expert medical opinion Short term disability and long term disability Life insurance and AD&D Supplemental life insurance (Employee/Spouse/Child) Health care and dependent care Flexible Spending Accounts Pre-tax commuter and parking benefits 401(k) Savings and Investment Plan with company match Paid time off: Flexible Vacation is provided to all eligible employees assigned to a salaried (non-overtime eligible) position. Accrued Vacation is provided to all other employees eligible for vacation benefits. For employees working at least 35 hours per week, the vacation accrual rate is 13 days annually for the first three years of employment and 18 days annually for subsequent years of employment. Vacation accrual is prorated for employees working between 20 and 34 hours per week. Employees working fewer than 20 hours per week are not eligible for vacation.
• Career Level - IC3 About Us Only Oracle brings together the data, infrastructure, applications, and expertise to power everything from industry innovations to life-saving care. And with AI embedded across our products and services, we help customers turn that promise into a better future for all. Discover your potential at a company leading the way in AI and cloud solutions that impact billions of lives. True innovation starts when everyone is empowered to contribute. That’s why we’re committed to growing a workforce that promotes opportunities for all with competitive benefits that support our people with flexible medical, life insurance, and retirement options. We also encourage employees to give back to their communities through our volunteer programs. We’re committed to including people with disabilities at all stages of the employment process. If you require accessibility assistance or accommodation for a disability at any point, let us know by emailing accommodation-request_mb@oracle.com or by calling 1-888-404-2494 in the United States. Oracle is an Equal Employment Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, national origin, sexual orientation, gender identity, disability and protected veterans’ status, or any other characteristic protected by law.