One of the key themes at the AI Engineer New York conference this week was how to balance agent autonomy with security and safety. Both frontier labs and financial firms focused on this issue in their presentations this week.
The main lesson: safe autonomy depends on where you put controls and human judgment, rather than how many restrictions you impose.
Sponsor: Build shopping into your agent with shop.dev
Adding commerce to an agent usually means rebuilding search and checkout from scratch. shop.dev gives you a single API for shopping in any agent or app. One connection gives you access to over a billion products with current price and availability, instant sign in with Shop, and one-tap Shop Pay checkout that happens right in the conversation. It’s open to any developer, with no account or key needed to try it. → Try shop.dev today
Joseph Kim, Head of ChatGPT for Financial Services at OpenAI, gave a keynote at AIE New York today. (We also did a special podcast with Kim, which will be released shortly.) He talked about reliability and a “threshold of usefulness,” which he said is “the point where a model becomes capable and reliable enough to save you time.”
Kim argued that ever-growing model capability will soon take financial analysts over the threshold and allow them to delegate more work to agents.
He made some specific predictions about AI’s role in finance in 2027. Not only will the financial industry rely less on Excel and PowerPoint next year, he claimed, but agents will be good enough to do “investment idea generation and trading.”
The “rising tide” of model capabilities is also leading to better traceability.
“In finance, you need to be able to check where a number came from and understand the context around it,” said Kim. “We wanted that review to be a natural part of the [ChatGPT for Financial Services] experience. These are the kind of product details that help turn a model’s capabilities into work that someone can use.”
As for when a human needs to be in the loop, Kim framed this as a product design decision.
“How would the interface change? What would the user still need to control, review, or approve? Those are product design questions you can start working on now.”
Balancing AI Safety and Innovation at a Hedge Fund
John Feminella, SVP Architecture at the hedge fund Two Sigma, is tackling the exact kinds of product design questions Kim referenced. In a keynote, he talked about five of the controls they’ve implemented to keep agents in check without limiting model capabilities.
Feminella argued that safety and innovation shouldn’t be viewed as opposite ends of a spectrum. He was especially critical of “approval fatigue,” where a person approving everything can hold back innovation — and also introduce risk, if they’re not fully paying attention to all the decisions they’re required to make.
At Two Sigma, they’ve implemented a set of controls, which he defined as “guardrails and shared capabilities that let teams move in ways that don’t hinder what they’re trying to do.”
The controls are composable, meaning “each one does one job and then together they let us run agents with real autonomy at an appropriate level of safety.”
One of the controls is to use auto-mode classifiers, which Feminella said “make decisions about routine calls and then send a person only the requests that are most important, the ones that are most likely to make a difference.”
Classifiers can be used for ambiguous cases, whereas deterministic rules are better for clear cases.
Anthropic: Give agents context, then define the boundaries of trust
Anthropic’s presentation outlined a similar framework, in its case with the goal of building “trust” into financial agentic workflows.
Last month, Anthropic announced Claude for Financial Advisors, allowing financial advisors to connect their systems to Claude and a set of workflow skills.
At AIE this week, Belinda Neal, Managing Director of Financial Services at Anthropic, explained why Anthropic has partnered with financial firms on the new produ