Datasets and expert raters for frontier AI training and evaluation
PlatinaData offers two core products: training and evaluation datasets, and expert raters for STEM, software engineering (SWE), medical and legal. Both offerings are differentiated by Calibrated Rubrics, Guided Corrections, and Verifier Benchmarks, turning domain expertise into measurable training and evaluation signals.
PlatinaData datasets have already reached frontier labs, delivered through tier-1 data organizations for models including Claude Opus and GPT. Partnering with frontier labs directly with our own RL platforms will help their researchers shape the tasks, rubrics, and failure targets from the first day of an engagement.
PlatinaData’s mission is to create richer datasets which reflect how experts think and solve tasks and not just the prompt where AI fails along with the corresponding right answer. This enables models to advance towards higher benchmarks with much lesser datasets.
PlatinaData’s core offerings
Training and evaluation assets
Project-specific training and evaluation datasets built around model failures, with original tasks, golden solutions, references, and expert review. Outputs include rubric-graded answers, corrected trajectories, process-reward labels, expert critiques, and held-out evaluations.
Engage the Expert Team
Domain-expert teams matched to each client’s required subdomains, credentials, and task specifications. Experts author tasks, assess responses, calibrate rubrics, guide corrections, and independently review outputs, supported by the MIDE (Machine Intelligence Domain Expert) program and automated validation.
Engage the team to build and continuously improve the assets
Domain coverage
Areas of deep expertise and specialist coverage
STEM
- Mathematics
- Algebra, geometry, calculus, probability, statistics, optimization, and numerical methods.
- Physics
- Quantum physics, classical mechanics, electromagnetism, thermodynamics, and condensed matter.
- Chemistry
- Organic, inorganic, physical, analytical, and computational chemistry.
- Biology
- Molecular and cell biology, genetics, biochemistry, neuroscience, and bioinformatics.
- Engineering
- Electrical, mechanical, chemical, and materials engineering.
SWE
- ML and AI development
- Model implementation, training, evaluation, fine-tuning, and ML pipelines.
- Data analysis and engineering
- Python and R, SQL, statistics, visualization, ETL, and analytics pipelines.
- Web and frontend
- JavaScript and TypeScript, UI frameworks, accessibility, and full-stack applications.
- Backend and infrastructure
- APIs, databases, distributed systems, cloud, DevOps, and performance optimization.
- Mobile, systems, and security
- Android and iOS, C and C++, embedded software, secure coding, and vulnerability remediation.
- Cross-cutting tasks
- Debugging, code repair, testing, refactoring, repository-level changes, and tool use.
Legal
- Legal research and reasoning
- Case law, statutory interpretation, citation verification, and jurisdiction-specific analysis.
- Commercial and corporate
- Contracts, transactions, corporate governance, and due diligence.
- Disputes and litigation
- Evidence review, discovery, procedural analysis, and legal drafting.
- Specialist practice
- Intellectual property, employment, competition and antitrust, tax, real estate, and financial services.
- Regulatory and compliance
- Privacy and data protection, cybersecurity, AI governance, AML and KYC, sanctions, anti-bribery, and sector-specific obligations.
Medical
- Diagnostics
- Radiology, pathology, laboratory medicine, and clinical reasoning.
- Biomedical and healthcare
- Clinical research, epidemiology, public health, medical documentation, and clinical informatics.
Engagement options
Datasets, expert raters, or a combined managed engagement. Specialist coverage and expert credentials are matched to the agreed project scope.
How the offerings are differentiated
The differentiators support both dataset delivery and expert rating. Each makes the resulting signal easier to inspect, reproduce, and use in training or evaluation.
Calibrated Rubrics
Experts review actual model responses before drafting criteria for must-haves, partial credit, and plausible errors. They grade each response and separately rank it holistically. Agreement between rubric scores and expert rankings reveals missing or mis-weighted criteria; independent expert grading establishes the human agreement baseline.
Dataset outputCriteria, expert grades, holistic rankings, and calibration evidence.
Expert-rater applicationA common scoring standard with measured agreement and adjudication of disagreements.
Guided Corrections
Experts intervene at the point where a model fails, using the smallest declared level of help: a nudge, concept, reference, or explicit correction. The model’s reasoning is preserved where possible, with intervention points and recovery outcomes recorded.
Dataset outputCorrected trajectories, process-reward labels, recovery labels, and expert critiques.
Expert-rater applicationStructured error diagnosis and consistent correction protocols that reveal how much help a model needs.
Verifier Benchmarks
Automated judges grade the same responses with expert grades hidden. Criterion-level comparisons measure false accepts and false rejects by domain and criterion type. Independent expert agreement provides a baseline for interpreting judge reliability, with remeasurement when judge models change.
Dataset outputBenchmark items, expert reference labels, and judge-reliability reports.
Expert-rater applicationIndependent grading and adjudication to validate automated scoring and expose systematic judge errors.
Quality controls and integration
The delivery workflow includes licence-cleared material, automated validation and near-duplicate checks, two-stage human review, and record-level provenance. Task specifications capture the model panel, validation results, and review history. Original tasks and separation of training and held-out evaluation material support contamination control.
We agree on the output schema and grading configuration with each client before production. Deliverables include the relevant rubrics, labels, references, and audit trail so grading can be rerun and quality assessed within the client’s workflow.
Success criteria
| Measure | Proposed evidence |
|---|---|
| Expert signal quality | Independent expert agreement, adjudication outcomes, and rubric alignment with holistic rankings. |
| Judge reliability | False-accept and false-reject rates by domain and criterion, measured against agreed thresholds. |
| Learning value | Repeatable model failures and recovery patterns; downstream gains measured by the client if training is in scope. |
| Delivery readiness | Acceptance rate, turnaround, schema compliance, reproducible grading, and complete provenance. |
Long-horizon computer tasks are coming soon.
This upcoming capability will extend the portfolio to multi-step workflows across applications, with reproducible environments, expert trajectories, and verifiable checkpoints. It is a roadmap extension; current engagements center on datasets and expert raters for STEM, SWE, and legal.
Work with us
Getting started: select a domain and engagement model, then agree the task specification, acceptance criteria, and delivery schedule.
Every dataset we deliver carries the evidence behind it: the calibrated rubric, measured expert agreement, and the error rates of any automated judge. Get in touch to see how that differs from standard expert data, and what it could add to your next training run.
References and prior art
The three differentiators follow established practice in frontier post-training. This section records the published work the approach draws on and the specific results that inform our design.
Rubrics as a delivered artifact
OpenAI’s HealthBench grades responses against physician-written rubrics at scale and reports clinician performance against the same rubrics, establishing a human baseline alongside the model score. OpenAI’s PaperBench develops its grading rubrics with the authors of each paper it replicates, breaking every task into individually gradable criteria scored by a model judge. Calibrated Rubrics follows this pattern, with the criteria drafted after the expert has read actual model responses.
Verifiable rewards
DeepSeek-AI’s DeepSeek-R1 and Lambert et al.’s Tülu 3 establish reinforcement learning against automatically checkable answers. For software engineering, SWE-bench (Jimenez et al., 2024) supplies the test-based pattern for repository-level tasks. Rubrics cover the reasoning that no program can settle, which is why both signals are delivered together.
Process and step-level supervision
Lightman et al., “Let’s Verify Step by Step”, shows that rewarding correct intermediate steps outperforms rewarding only the final answer. Guided Corrections produces that signal directly: the intervention point, the declared level of help, and whether the model recovered become process-reward and recovery labels on the trajectory.
Judge reliability
Zheng et al. document how closely an LLM judge tracks human preference and the position, verbosity, and self-preference biases that come with it. Verifier Benchmarks treats that reliability as a measured quantity: judges grade with expert labels hidden, false accepts and false rejects are reported by domain and criterion type, and independent expert agreement provides the baseline against which a judge score is interpreted.
Rubric and environment infrastructure
WebArena (Zhou et al., Carnegie Mellon University, 2024) and OSWorld (Xie et al., The University of Hong Kong and collaborators, 2024) set out the anatomy of a reproducible agent environment: self-contained applications reset to a known initial state, a task collection, and an execution-based reward that reads the final state. Where no program can decide the outcome, a rubric supplies the reward instead. The agreed output schema is designed to populate those fields, so records and rubrics can be loaded into an existing training stack without reformatting.
Long-horizon computer use
OSWorld 2.0 (XLANG Lab and collaborators, arXiv:2606.29537, July 2026) scores roughly 27 checkpoints across 108 long-horizon tasks. The strongest agent reported reached 83.5% on the shorter OSWorld-Verified tasks but completed 20.6% of the long-horizon set. That gap is what the roadmap extension targets, with checkpoint-level verification rather than a single end-state check.
Figures quoted above are from the cited sources. Published results move quickly; each engagement reports its own measurements against expert labels collected during the engagement.