Can a company license evaluation datasets to AI labs?
Yes, when the questions come from real work and the answers were checked by people who know the job. Labs test their models on sets of tasks with known correct answers. Public test sets leak into training data and lose their value, so labs look for private ones. A company that grades work, trains staff or checks quality often already holds questions with verified answers.
Where evaluation material hides in a company
| Certification and training exams | Questions staff must pass before they can do a job, with the answer key and the reason each answer is right. |
| Graded work samples | Real tasks scored by a reviewer against a rubric: a coded claim, a support reply, an estimate. |
| Test cases for software | Inputs and expected outputs written by a QA team over years of releases. |
| Expert second opinions | Cases where two specialists reviewed the same file and their agreement or disagreement was recorded. |
| Audit and quality samples | A batch of finished work pulled for review, with the errors the reviewer found. |
What makes an evaluation set useful
Each item needs a clear task, a correct answer that someone qualified checked, and ideally the reasoning or rubric. Hard items matter more than easy ones: the cases where trained staff disagreed or where a first answer was corrected.
Size matters less than with training data. A few hundred well-checked, hard tasks from one profession can be worth more than many thousands of easy ones.
Why it has to stay private
Once a test set is posted online it ends up in training data, and a model that has seen the answers scores well without learning the skill. That is called benchmark contamination. A buyer will usually ask that the set has never been published and will ask for exclusivity, so the questions stay unseen by other labs.
Keep a record of where each item came from and who checked it. That history is what lets a buyer trust the answers.
Preparing the set
Take personal and client details out of each task while keeping it realistic. A support question still needs the product, the symptom and the steps already tried. Rewrite only what identifies someone.
Describe the set in a short sheet: task types, count, how answers were checked, who checked them and how often reviewers agreed.
What does not belong in an evaluation set
| Questions copied from public sources | Textbook or online questions are already in training data and add no value. |
| Answers nobody checked | An unverified answer key misleads the buyer and is left out. |
| Licensed exam content | Questions from a certification body or vendor belong to them. |
| Tasks that cannot be cleaned without losing the point | If removing client details makes the task meaningless, it is dropped. |
More on this: how personal details are removed, who carries the risk and how you get paid.
An example, invented for illustration
A medical billing company with seventy coders runs a monthly quality check: a reviewer recodes a sample of claims and records each disagreement with the reason. Over five years that adds up to thousands of claims with a checked code and a written reason. Patient details are removed, the hardest disagreements are pulled into a set of several hundred tasks, and a sheet explains the review process.
Questions owners ask about this
How is an evaluation set different from training data?
A lab trains on one and tests on the other. Test items have to stay unseen, so they are licensed under tighter terms and are usually smaller, harder and better checked.
Our internal exams are old. Are they still useful?
If the work has not changed much, yes. Mark the date and the rules each exam follows, since a question tied to a rule that changed may need a new answer.
Do we lose the set once it is licensed?
That depends on the terms. Many evaluation licenses are exclusive and forbid publishing the items, so the company can keep using them internally but not sell them again.
Didn't find your question? Ask us directly.
Keep reading
Related kinds of records
- Labeled tables and datasets: Rows, a result column and a data dictionary: when a table can be licensed.
- Preference and rating data: Scores, edits and A or B choices made by staff at work, linked to the item judged, which labs use to tune models.
- Agent trajectories: Step-by-step task paths from audit logs, ticket histories and automation runs, the record AI agents learn from.
Industries that keep these records
- AI product companies: Evaluation sets, rating records, support data and failure notes; user prompts, customer content and restricted model outputs stay out.