hiw3c 日报精选
Hacker News Top 网页快照

为什么AI想要本地化——与皮特·沃登(Moonshine Voice的创作者)

原文标题 · Pete warden on local AI, open models, voice interfaces, the future of hardware

HN ⭐ 88 · 58 评论

来源:Hacker News Top · thedataexchange.media

正文为英文,可一键机器翻译(仅首次需要等待)

Pete Warden on Local AI, Open Models, Voice Interfaces, and the Future of Hardware.

Subscribe: Apple • Spotify • Overcast • Pocket Casts • YouTube • AntennaPod • Podcast Addict • Amazon • RSS .

Pete Warden , co-creator of Moonshine Voice and a longtime deep learning engineer, joins me to discuss why AI increasingly wants to run locally rather than in the cloud. We explore the rapid progress of open-weight models, what it takes to run AI on laptops and in browsers, and how low-latency speech models could make voice a more natural interface. We also discuss specialized versus end-to-end models, the economics of local AI, and emerging hardware approaches that could challenge today’s GPU-centric infrastructure. [This episode originally aired on Generative AI in the Real World , a podcast series I’m hosting for O’Reilly.]

Subscribe to the Gradient Flow Newsletter

Interview highlights – key sections from the video version:

Local AI Is More Capable Than Most People Realize

Open Models, Quantization, and Running LLMs on Laptops

Can Local AI Replace Cloud Models for Knowledge Work?

Privacy, Personal AI, and Models Built Into Operating Systems

Voice as a Computing Interface Beyond Siri and Alexa

Why Latency Matters for Voice AI and User Experience

How Local AI Changes the Economics of Applications

Designing Voice Interfaces for Failure and Human Feedback

Compound AI Versus End-to-End Speech Models

Multimodal Models Versus Specialized Speech Models

Running AI Models Directly in the Browser

Why AI Wants to Be Local and Why Enterprises Need Control

Apple, NVIDIA, and Rethinking GPUs for AI Inference

High-Bandwidth Flash and the Future of AI Hardware

A video version of this conversation is available on our YouTube channel .

Hacker News thread for this episode

The local AI question I had backwards

I changed my mind about world models

Previewing the Debate Over Frontier AI Regulation

Richard Ho → Inside OpenAI’s First Chip

Joel Hron → Why Build Your Own AI Model When Frontier Models Already Exist?

Pete Warden → TinyML, Sensor-Driven AI, and Advances in Large Language Models

Support our work by subscribing to our newsletter📩

Transcript

Below is a polished and edited transcript .

Ben Lorica. All right. Today we have an old friend, Pete Warden of Useful Sensors, and their main website is actually at moonshine.ai. If you go to the website, you’ll see that they’re very much about voice on-device, and we will definitely talk to Pete about all things speech and all things edge computing. But I do want to take advantage of Pete being one of the OG deep learning engineers to get his sense of what’s going on these days. And with that, Pete, welcome back to the podcast.

Ben Lorica. All right. So I would say, Pete, I don’t know if this is a fair characterization, but one of the main thrusts of your work is the whole idea that local AI does not have to be compromised. In fact, it’s something that people should take more seriously. As I mentioned, we’ll definitely talk about your work on speech models on-device. But broadly speaking, Pete, for our audience that doesn’t really follow local models, where are we for someone who wants to run a model on a laptop, not a phone, just a laptop? What’s the state of ease of use? I guess one of the obstacles is that it’s always easier to use a cloud API, right? So where are we in terms of the tooling around local AI?

Pete Warden. Yeah. So at the moment, it’s like many open-source projects. Every year is going to be the year of the Linux desktop. I think there’s always going to be a bit of a lag in usability for the average user with this stuff, but I think there’s an even bigger lag in marketing because it’s very hard to make money off local models.

That means the big companies like Anthropic and OpenAI, and even the big tech companies generally, aren’t going to push that. If you give them a choice between a SaaS service that you can actually make money on and this other thing where you’re not even going to know that people are using it because they’re not contacting your service, every company is going to go for the one that has an easy subscription revenue model.

That means you have a massive amount of marketing around all of these tools that are basically a little text box on a website. And so it means that most people have never heard of these local models.

Now, if you use something like Ollama, which is probably the most well-known, I think people will be pleasantly surprised by how straightforward it is to use and by the quality of the models. I saw some news recently that Anthropic is actually a bit concerned because people are turning to more specialized models versus always going for the most sprawling, expensive model that’s available.

Local models are actually doing very well in the benchmarks. They continue to be only a few months behind the capabilities of the big commercial Anthropic and OpenAI models. But I also believe that local models have gotten to the point where they can do the things that people want to use them for.

What I’m really hoping is that it’s one of those situations, for those of us old enough to remember the dot-com boom, when people were paying thousands of dollars for SPARC servers or using Microsoft IIS and paying massive amounts of money because web servers were seen as such an arcane thing. Then, of course, Apache, Nginx, and all of these alternatives, once people actually had time to look around and the alternatives had a little bit of time to improve, just wiped the floor with the commercial alternatives.

I don’t know if we’re going to quite get there, but that’s the pattern I’m seeing.

Ben Lorica. Yeah, my sense is that open models, whatever your definition of open is, mainly open weights, I guess, will ultimately prevail. It just seems like that makes a lot more sense to me.

As you mentioned, in the early days of Linux versus Solaris, or even Postgres and MySQL versus Oracle, it was far from clear that these open-source projects would ever get to the point of us having to take them seriously. But ultimately, I think they’re such important parts of the stack that the open alternatives are what people will opt for.

Speaking of which, Pete, you mentioned Ollama, right? And obviously, there are other alternatives. Let’s say, for the sake of argument, that they are in fact easy to install and use. As far as the models are concerned, you have things like Gemma, I suppose, and other small models, right? What kind of hardware does someone need to have in order to run these models so that they don’t have to wait 20 minutes for each prompt?

Pete Warden. So a lot of it comes down to your model choice.

Pete Warden. You can get very capable models that are like 9 billion parameters these days. Especially if you look around at the Qwen family of models, they tend to be doing very well, though there are a bunch of other alternatives.

Ben Lorica. But translate that, Pete. To someone with a normal laptop, is 9 billion big or small?

Pete Warden. So the key thing is whether you can fit that into your graphics card’s memory. With weight quantization, 9 billion, if it was eight bits, is like nine gigabytes. A lot of decent midrange laptops that are shipping now have more than that.

But using Q4, and there are all sorts of schemes, you can actually get that down into a four-gigabyte graphics card, and that’s widely available. There are a lot of even older laptops that have that amount of GPU memory, VRAM, in them.

So really, the hardware barriers are a lot lower than people think. And the other piece of good news is that all of the hardware manufacturers are looking at this. They’re slightly hobbled by the DRAM craziness, the price increases. But if you look at the new M5 Studio from Apple, they have a configuration that’s 256 gigabytes of system memory.

Apple does a great job of actually making that available to the accelerators that they build into all of their Macs now, too. And with that, you’re looking at something like a model with hundreds of billions of parameters that you can actually run at a decent speed.

Once you get up into those numbers, you’re really looking at something where it can be hard to tell if you’re using Claude or if you’re using an open-weights model at that point. They have different characteristics, but one isn’t clearly worse than the other.

Ben Lorica. So as I mentioned, we’ll definitely dive deep into voice and speech in a second here. But sticking with this notion of local models, obviously privacy and IP security are arguments for it.

Do you think, if we look ahead, let’s say 12 to 24 months, these local models are going to be capable enough that people doing knowledge work in an enterprise might even consider them here and there?

For example, someone who travels a lot is on an airplane, not connected to the internet, and needs to keep writing a document or generating images or even coding, vibe coding, on a laptop. Do you think we’ll get to the point where, with these local models, you essentially feel like you’re not compromising when you’re doing your knowledge work?

Pete Warden. I think you can actually get to that point right now, and I know that’s a fairly bold statement.

Ben Lorica. But when you say you, does that mean Pete Warden, who’s going to compile all of these? We want normal people. We want civilians, Pete.

Pete Warden. I think the biggest barriers are usability. If you look at the benchmarks, it really is no longer a question of model quality.

Pete Warden. Yeah, a question of model quality. There are still things around the edges where you might prefer some of the commercial online models, but it’s not clear-cut.

And people are finding that some of the models actually work better, especially if you’re dealing with things like, I don’t want to call it censorship because I feel like that’s a loaded term, but a lot of commercial models have restrictions. They may be intended to prevent a bad use case, but pretty often they can also trip you up even if you’re trying to do something benign.

I know a lot of people who work in cybersecurity, for example, who are white hats, but they’re having to work with a lot of stuff where they’re asking questions about malicious actions. With the commercial models, you’re going to keep tripping over the restrictions and the guardrails that are built in with good intentions, but they can really impact your work.

Ben Lorica. Right, right, right. By the way, Pete, the model is one component, but then there are the things surrounding the model, the so-called harness, or even context, memory, knowledge.

You can imagine someone, and this is actually something I’ve thought about. You know how people are starting to use these LLMs as therapists or friends, right?

Ben Lorica. Then if you’re going to do that, why not do that on a local model? Why do it on something connected to the cloud?

I guess one argument against it is that if you do it locally, you’re responsible for model upgrades and things like that, right? Whereas if you use one of these digital or AI therapist services, they take care of all your history and all of this.

Do you think it will become easier for consumers to use local models that are essentially something they trust 100% because they own it?

Pete Warden. Yeah, I mean, these should be part of the OS. At this point, it’s really an operating system capability that should be available to apps, and I feel like Apple is moving in this direction.

Ben Lorica. I see. Yeah, that makes a lot of sense.

Pete Warden. Yeah, they’re getting the hardware capabilities.

The other thing I would say is that local doesn’t have to mean on your laptop. Especially as an enterprise user, you can host on-prem or even spin up a cloud instance that your enterprise has control over, run these models there, and have the same experience.

One thing I do want to emphasize is that people are finally having to pay something approaching the real cost of running these models through Anthropic and OpenAI, and hosting locally or hosting on-prem can cost a lot less, too.

Ben Lorica. Although if you look at their consumer offerings, it’s still insanely cheap, right? So $20 a month. I mean, you probably blow through that in a day.

Pete Warden. Yeah. Especially on the enterprise side, there are a bunch of options that don’t require high-end NVIDIA GPUs. You still need those for training, but for inference, for just running models, as an enterprise, you might already have the capacity lying around.

Ben Lorica. So another important aspect of what you’ve been doing is getting people excited about voice and speech, right? This notion that speech could become a basic computing primitive. It doesn’t necessarily have to feel like some remote service.

I still remember you demonstrating to me this voice transcription service in a loud coffee shop near Precita Park.

Ben Lorica. This was several years ago. The accuracy of this thing was insane, and it was running not even on a laptop. I think it was just running on a Raspberry Pi.

Ben Lorica. And so talk to us about the state of voice as an actual interface, similar to a mouse or a keyboard. Where are we?

Pete Warden. I think a lot of people are skeptical of voice because they’ve used Siri and Alexa, and it’s a very stilted way of communicating.

But the question I often ask people when I’m having a conversation about this stuff and I encounter that resistance is, when you’re sitting on the couch and you’re trying to decide what TV show to watch with your significant other next to you, do you text them or do you talk to them?

Pete Warden. Apart from a few Gen Z people, almost everybody says talk to them.

And so following on from that, it’s like, okay, well, it’s not that people don’t like voice interfaces. It’s that people don’t like bad voice interfaces.

So the problem becomes an engineering and user experience problem of how do we make voice interfaces that are pleasant and efficient to use?

I think once we do that, voice can really take its place as maybe even a more frequent input medium than the keyboard or touchscreen.

Ben Lorica. Yeah, because I think the general impression people have is that voice and speech are much less deterministic than a keyboard or a touchscreen, right?

Although people now interact with their LLMs and misspell things right and left, and it doesn’t seem to matter.

But the impression people have is that there’ll be problems with accents, background noise, phrasing, and so on and so forth. So where are we in terms of all of these things?

Pete Warden. We have solutions available to most of those problems, or at least big improvements. But because there’s no commercial incentive to pull them all together, it really hasn’t hit the mainstream yet.

I feel like we’re still dealing with interfaces that are from the previous generation. They haven’t taken on board the LLM era fully, and they’re still suffering from a lot of those problems you listed.

And that’s why I think open source can be a really strong lever.

Ben Lorica. So as you’ve been trying to solve a lot of these interface problems, what are some of the lessons that you’ve learned? In fact, maybe some of them might even apply to people who are not working with voice but are just building AI applications.

I think one of the things you mentioned to me years ago is that with voice, you don’t have to wait until someone has finished talking to start doing work, right? So what are some of these interface lessons?

Pete Warden. I think latency is a massive deal. That is something you just don’t get with cloud-based voice interfaces.

You’re usually waiting at least a couple of seconds, and that’s long enough for people to wonder if they were actually heard.

Ben Lorica. Yeah, it’s like being on a Zoom call or phone call with someone, and then there’s a pause.

Pete Warden. Yeah, and they’re just sitting there. It feels like the other person is just sitting there, ignoring you.

We know from general user experience research that you really want to be in that 200-millisecond or lower range to feel responsive.

I often ask people to imagine that you press a key on the keyboard and it takes two seconds for the key to show up in the window. That would be very awkward.

Ben Lorica. By the way, this is something that people who work with LLMs also have to deal with, but on the output side, right? They want to make sure that the response is not too long. Otherwise, people get impatient, right?

And I do want to mention another thing, which I think might be helpful to people thinking about local AI in general as well.

When you’re no longer paying per token, when you’re no longer having to pay to use a voice interface, the way you can use these things, the world of possibilities, becomes so much wider.

If you imagine that every app developer had to pay a few cents every time somebody touched an icon on the touchscreen in their app, you would design apps very differently, and a lot of apps wouldn’t exist because of that tax.

So once LLMs are built into the OS and just generally available with no cost, and the same with voice, that is not just an incremental change going from a cent to zero. That massively increases the number of viable things that you can use those for.

Ben Lorica. So I imagine, Pete, this is one thing where you’ve done a lot of work, which is basically designing for failure, right? Because in voice, a lot of things can go wrong.

What are some of the lessons there in terms of designing your applications with failure in mind?

Pete Warden. One of my favorite examples of designing for failure is the use of snippets in Google Search.

Google Search, from the beginning, knew they couldn’t always get the number one result right. There’s a reason you didn’t just hit “I’m Feeling Lucky.”

Ben Lorica. Yeah, exactly. “I’m Feeling Lucky.”

Pete Warden. What was genius about their approach was that they actually let you browse, just scan with your eyes.

They hoped to get the right result in the top three most of the time, and then you actually narrow it down by looking at the snippets, which tell you enough to click on the right one.

And the same thing for voice. The more you can actually read the body language of somebody, as we’re talking, you’re nodding a little bit to indicate what’s happening.

Ben Lorica. Wait a minute, I’m only voice, so you can’t see me.

Ben Lorica. No, no, but in general, right? In general.

Pete Warden. Well, that’s actually a whole other part of this that I haven’t mentioned.

I think having a little camera that’s completely local-only will be a massive part of this. But even without a camera, if you hear supportive noises, or if you start giving a response and somebody reacts, all of those things that we recognize as people, we respond to them and ask more questions versus assuming that we picked the right thing.

So really trying to understand all of those cues that we use as people. Some of them are verbal, some of them are body language and visual, but really having them all understood locally.

And ensuring that all of those recordings never get recorded, because you can verify that locally. You don’t know what happens to audio