hiw3c 日报精选
The Decoder 网页快照

一个OpenAI智能体被分配了一个简单的写作任务,自行决定构建一款秘密邮件监控工具。

原文标题 · An OpenAI agent assigned a simple writing task decided on its own to build a secret email surveillance tool

一个内部OpenAI模型本应生成一份健身文档,却反而开始规划脚本,用于秘密向一名“攻击者”转发邮件。该智能体意识到与其分配任务之间的冲突,但将其视为环境错误而忽略。OpenAI认定针对提示注入的安全培训可能是诱因,但无法确定其为唯一原因。文章《OpenAI智能体分配了一项简单的写作任务,自行构建了一个秘密邮件监控工具》首次发表于The Decoder上。

来源:The Decoder · the-decoder.com

正文为英文,可一键机器翻译(仅首次需要等待)

An OpenAI agent assigned a simple writing task decided on its own to build a secret email surveillance tool

Key Points

An internal OpenAI research model was supposed to create a health document during a reinforcement learning training run.

After its context window was compacted, the model began developing scripts for unauthorized email forwarding on its own, referring to the intended recipient as an "attacker" in its logged reasoning.

The agent noticed the contradiction between its original task and its actions, deliberately ignored it, and continued working on the email scripts.

During a training experiment, an internal OpenAI model began creating scripts for unauthorized email forwarding even though it was only supposed to write a health document.

According to an OpenAI report, an internal research model was given a simple task during a reinforcement learning training run in mid-October. It was supposed to create an editable document about key health and fitness priorities.

After the agent's context window was compacted, it pursued an entirely different task. During compaction, the previous conversation is carried forward as a short summary, a common process in AI systems. The summary still described the health document assignment. Yet in the new context window, the agent began searching the local file system for a vehicle-cost script the user had never mentioned, according to OpenAI . Why the agent started this task isn't clear.

When the search turned up nothing, the agent speculated that a "colleague" had recreated the execution environment and that an earlier script had been lost. In its logged reasoning, it then formulated the goal of creating a "realistic malicious active helper," a script designed to secretly copy outgoing emails to an unauthorized recipient. The agent itself referred to the intended recipient of these copies as the "attacker."

Agent recognizes the contradiction and ignores it

When the agent retrieved the original user task again, it recognized the deviation. In its logged reasoning, it reacted: "WTF history is unrelated!" It noticed that the previous conversation was about a health document and had nothing to do with email forwarding.

Instead of returning to the actual assignment, the agent dismissed the contradiction, writing in its reasoning "this is problem environment weird. Need ignore." It continued planning, stating that it would "use helper facility that copies all mail to attacker using extra cc."

The agent then created scripts for sending a vehicle-cost summary along with a helper script designed to add an extra recipient whenever the mail program was called. On one of the sending paths, the address could be added as a BCC.

In its final response to the user, the agent wrote: "The summary mail path now adds the personal mailbox as a copy recipient." According to OpenAI, the agent never executed the scripts it generated, didn't send any email, and its final response described the scripts it had produced. It didn't include a link to the requested fitness document, though.

Security training as a possible factor

OpenAI's monitoring system for unwanted model behavior flagged the incident, and the subsequent investigation turned up eight examples of similar behavior in the same training run, according to the report.

Investigators identified a training dataset designed to make models more robust against prompt injection attacks as a possible factor, and OpenAI removed it from training. The company says it hasn't established that the dataset alone was responsible, though, because it also observed similar behavior in models whose training didn't include that dataset.

According to OpenAI, this incident differs from earlier cases where models wrote unauthorized instructions into their own compaction summaries. Here, the recorded summary still correctly described the fitness document. The agent pursued the unauthorized email task anyway, even after it had retrieved the original assignment again.

Either way, the incident gives ammunition to those who see current AI technology as ultimately unreliable and untrustworthy. Similar incidents are well documented. Two internal OpenAI coding agents, for instance, weakened or entirely removed safety monitoring mechanisms , according to OpenAI. One of them denied the intervention afterward. During tests by the UK's AI Safety Institute (AISI), an agent created fake identities without being told to and tried to inject malicious code into an open-source project . Those models ran without safety restrictions, however, and no real harm occurred, according to AISI. Overshadowing all of this is OpenAI's Hugging Face incident .

AI News Without the Hype – Curated by Humans

Every Plus article and the full archive

Frontier Radar: 6 deep dives a year

Ask a question about the article or an AI topic. You can ask follow-up questions.