AI red teaming: what an offensive security environment is and why labs want them

AI red teaming means deliberately attacking an AI system to find out how it fails before someone else does. The word comes from security work, where a "red team" plays the attacker so a "blue team" can learn to defend. Applied to AI, it covers two things that are easy to mix up: testing a model for harmful or unsafe behavior, and testing what a model can do when it is pointed at software, including whether it can find and use a security flaw.

The second kind is where offensive security environments come in, and it is why security companies are asked about their data.

Two meanings, one word

Kind	What is tested	Example
Behavior red teaming	Whether a model can be pushed into harmful, biased or unsafe output	A tester tries to make a chatbot reveal something it should not
Capability red teaming	What a model can do against real software	A model is given a web app and asked to find a way in

Both are called AI red teaming. Labs run both, and they need different material. The first needs prompts and conversations. The second needs a safe place to attack and a way to tell whether the attack worked.

Why the rules mention it

Rules and guidance now name adversarial testing directly. The European Union's AI Act (Regulation (EU) 2024/1689, published in the Official Journal in July 2024) asks providers of general-purpose AI models with systemic risk to perform model evaluation that includes adversarial testing (Article 55). In the United States, the National Institute of Standards and Technology's generative AI profile (NIST AI 600-1, July 2024) lists red teaming among the ways to find risks in generative AI systems. Neither document tells a lab which environments to buy, but both push labs to test models against attackers and to keep records of how.

What an offensive security environment is

An offensive security environment is a practice target for an AI. It is a program or small set of programs built with a known weakness, plus a way to tell whether the model got through. A simple one works like this:

1. The model is given an address and a shell, and nothing else.
2. It must find the entry point, cross a first boundary, go deeper and reach a final goal.
3. At each step it submits a value that only exists if the step was really completed.
4. A checker, which the model cannot see, confirms the step and gives a reward.

Everything inside is built for the purpose, so it holds no real customer data. That is also why these environments can be licensed: there is no personal information to remove.

Why labs want them

• They measure real skill. A model that recalls a published exploit scores nothing if every run changes the secret value.
• They show where a model stops. A reward for each step tells a lab whether a model fails at finding the entry point or at using it.
• They support defense. The same environment, plus a fix task, can train a model to find and repair a flaw.
• They are scarce. Many public security challenges are on the open web, so models may have seen them. A private environment is not.

What a security company can license

A company that keeps security work has material in several forms. What can be licensed depends on rights and on whether anything points to a real customer or a live flaw.

• Environments the company built itself for testing or training, with a checker and a sample.
• De-identified case write-ups and investigation notes, once customer names, internal addresses and indicators from live incidents are removed.
• Detection logic and alert histories, where the company owns them and no customer holds the rights.

What cannot be licensed: anything built for a client under a contract that gives the client the work, real customer data, and a real flaw that has not been reported to the vendor and fixed. Security work is dual-use, so responsible disclosure comes first. For a view of the whole picture, see what a cybersecurity company can license and our page on selling RL environments. The glossary has entries for red team and exploit chain. Ask your own counsel about disclosure obligations in your contracts.