Gray Swan, AI Red-Teaming Startup Founded by Ex-CMU Professors, Raises $40M Series A

Gray Swan raised $40 million in Series A funding co-led by Wing VC and Madrona, bringing the Pittsburgh-based company’s valuation to $200 million. The round included participation from Snowflake Ventures, Hudson River Trading, and Samsung Next. Gray Swan was founded in 2023 by Matt Fredrikson and Zico Kolter, both former Carnegie Mellon University professors.

The company works with OpenAI, Anthropic, Google Deepmind, Meta, xAI, and ByteDance. Its technology has been cited in 11 frontier model system cards, including those for GPT-5 and Mythos. OpenAI became Gray Swan’s first customer less than a month after the company launched, using its tools to jailbreak the o1 model family.

Gray Swan operates Arena, a platform used by approximately 15,000 security professionals worldwide to test AI systems. Kameron Bettridge, a 23-year-old security engineer at Blizzard Entertainment, has participated in more than 1,000 challenges on the platform and earned $10,000. The company currently has 20 enterprise customers.

Gray Swan has developed Shade, an AI agent that searches for vulnerabilities, and Cygnal, software that monitors model prompts and outputs. Data collected through Arena trains these tools. Zico Kolter, chief scientist and cofounder, sits on the OpenAI Foundation board as chair of its safety and security committee.

In 2023, Fredrikson and Kolter discovered a jailbreak method involving a string of random characters that bypassed safety filters on models from OpenAI, Anthropic, Meta, and Google. The technique was later fixed.

Snowflake uses Gray Swan’s software to test its coding agent Cortex Code and its general-purpose agent Snowflake Intelligence. Anupam Datta, principal research scientist at Snowflake, said the tools help detect malicious prompts that could instruct agents to send proprietary data to adversaries. Fredrikson, now CEO, said agents are creating new attack surfaces that are not yet obvious.

Kolter noted that agents are actively looking for prompt injections rather than stumbling upon them. The company plans to use the new capital to expand sales of its security tools to more enterprises building AI agents.