How is an RL environment graded without an AI judge?

An RL environment is a task, a world to act in, and a checker that scores the result. The checker is the part that decides what an environment is worth. When the checker is a short program that looks at what really changed in the world, the score is called a verifiable reward. When the checker is another AI reading the answer and giving an opinion, it is called an AI judge. Labs increasingly ask for the first kind, and the reasons are practical.

What a verifiable reward looks like

Take a task such as "reset this user's password in the admin tool." A verifier does not read the agent's explanation. It opens the system afterward and asks one question: is the password for that user different from before, and does the old one fail? The answer is yes or no, and the program that asks it is the reward.

The same idea applies in other fields:

• Coding: the tests pass or they do not.
• Math: the final answer matches a known value.
• Security: a secret value only reachable by exploiting the flaw was returned, and a log shows the matching action.
• Operations: the ticket reached the right state and the refund was recorded once.

None of these needs an opinion. That is the point.

Why labs prefer it to an AI judge

An AI judge is cheap to set up, but it brings its own weaknesses into the training loop.

1. It can be fooled. A model being trained learns what the judge likes. If the judge likes confident, well-formatted answers, the model produces those, whether or not the task was done. This is often called reward hacking.
2. It is not stable. The same answer can score differently on a different day or with a different judge model, which makes progress hard to measure.
3. It is hard to audit. A buyer who is asked to pay for an environment wants to see why a score was given. A program can be read. An opinion cannot.

Research on training language models with checkable rewards has made the approach well known. A November 2024 paper from the Allen Institute for AI, led by Nathan Lambert and posted on arXiv, describes training with verifiable rewards on tasks such as math and instruction following, where a program can decide if the answer is right. The lesson for a builder is simple: where a task can be checked by a program, labs would rather use the program.

What makes a checker good

A checker is only as useful as it is hard to cheat. Labs and their evaluation teams look for:

• A check on state, not on text. The verifier inspects what changed, not what the agent said it did.
• A secret the agent cannot guess. In some security tasks the checker looks for a value that changes on every run, so a memorized answer scores nothing.
• A clear pass for each step. A task with several steps can award a small reward for each one, which gives a richer signal than pass or fail.
• A twin that should score zero. Some builders add a safe copy of the task where a reported flaw must score nothing, which measures false alarms.
• No access to the answer. The model never sees the checker or its logs.

What this means if you build environments

If you built a task and a checker, you may hold something a lab will license. Before you talk to anyone, write down:

• how many tasks you have and how many kinds;
• how each one is graded and whether any part uses an AI judge;
• how hard the tasks are and how you measured it;
• whether they come with seeded variations, a datasheet and a sample;
• who owns them and whether any third-party or client code is inside.

Our page on selling RL environments explains what labs look for and what cannot be licensed. The glossary has short entries for verifier and reward. If you are not sure whether what you have is worth licensing, the data value calculator takes about a minute, and environments are priced one at a time.