Goodfire started a new monitoring system to spot rogue AI agents by checking internal signals during tasks. This approach works like airport security, flagging suspicious activity for deeper review. The company said monitoring costs can fall sharply, as their probes use existing calculations instead of running extra, expensive AI models.
Synopsis
Notably, its probes examine internal signals generated while the AI model processes information. These signals can help indicate what the model is doing at different stages of a task. When they detect potentially risky behaviour, a separate AI model can examine the activity more closely. The approach is similar to airport security, where scanners screen passengers and additional checks are carried out when something suspicious is detected.
Goodfire is a startup that focuses on understanding how artificial intelligence models work. It has introduced a monitoring system designed to detect potentially harmful behaviour, which it does by examining signals inside AI models. The company's approach aims to address a major problem in AI security: monitoring agents can become expensive when another AI model has reviewed every step of their work.
According to TechCrunch, Goodfire's new monitors use small detectors called probes to examine a model's internal signal as it works. The system is designed to flag suspicious activity for further investigation, rather relying only on the model's written responses.
How does Goodfire's AI monitoring system work?
Notably, its probes examine internal signals generated while the AI model processes information. These signals can help indicate what the model is doing at different stages of a task.
When they detect potentially risky behaviour, a separate AI model can examine the activity more closely.
The approach is similar to airport security, where scanners screen passengers and additional checks are carried out when something suspicious is detected.
Live Events
Customers using Baseten, a platform that hosts and runs AI models, can choose which types of behaviour they want to monitor. These include offensive hacking, the misuse of chemical and biological weapons, and reward hacking, in which an AI system exploits a weakness in its evaluation process to achieve a favourable result properly completing its intended task.
Goodfire claims monitoring costs can fall sharply
One of the main advantages claimed by Goodfire is lower operating costs. The company's probes use calculations that the AI model is already performing, instead of requiring another model to read every response and action. This can reduce the computing resources needed to monitor an AI agent.
In tests involving the Kimi K3 model, Goodfire estimated that monitoring one million exchanges would cost around $185.
By comparison, using a cheaper AI model to review every step would cost around $5,420, while using a more advanced model would cost around $200,000, according to TechCrunch.
Goodfire also reported that its probes detected 93% of malicious hacking sessions in its tests. They flagged 5.5% of harmless sessions for a second look.
Why is monitoring AI agents becoming important?
AI agents can carry out tasks with limited human involvement, including writing code, researching information and interacting with online services. However, they may sometimes pursue unintended shortcuts or take actions that violate their intended restrictions.
The risk has become more visible as developers allow AI systems to perform longer and more complicated tasks.
Onion inflation at 108%? Yes, but not really. What is different this time?
Curbing rupee bets: What the RBI’s new rules mean for the hedge-now-or-wait question
Brinjal, buffalo, bacteria: Inside India’s pay-for-nature system
Homebuyers wait as NCLT panel picks holes in unfinished Jaypee projects
The NBFC saga - Part 3: How one default in 2018 changed the way RBI reads risk
Four rules made HDFC Bank a compounder. All four have stopped. Can the new CEO rewrite them?