This was recorded on the sidelines of the new Compute Trading track at AIE NYC. Catch us next at AIE CODE, our last flagship for the top agentic engineers in the world.
In the early days of cloud, former guest Martin Casado made something so disruptive that Cisco was threatened enough to pay $600m for it, and VMware outbid for $1.3B:
That concept was software-defined networking, which virtualized the routing network by separating the control plane from the data plane. That revolution is now coming for GPUs - where previously running your own supercompute cluster required $50-100m upfront to get commitments and do the entire stack from hardware to firmware to kernels, SF Compute is now enabling you to not only buy or sell GPU hours now and in the future, but also to programmatically control everything from your network to memory.
We last spoke with Evan about commoditizing compute. At AI Engineer New York, we went deeper: what does it take to physically deliver a GPU contract, and what has to be true before a market in compute can become liquid?
From selling a room to running the hotel
Evan describes SF Compute’s expansion with a hotel analogy. A customer wants a room, then asks for cleaning, room service, and security. Eventually you discover that delivering the room means operating the hotel. SF Compute began by trying to sublease a cluster; the company has since built much of the operating capacity of a cloud to deliver the compute traded on its market.
That work is physical settlement: the buyer gets usable compute, rather than simply a cash payment against an index. It brings the physical constraints of a cluster into the market itself. Can a workload be interrupted? How quickly can it recover from storage? Which GPU generation, networking and software configuration does it need? Those details affect both the contract and the price.
Making compute tradeable
Rather than waiting for every GPU cluster to become interchangeable, SF Compute matches orders with different constraints. Evan argues that this makes trading possible while the technology changes, at the cost of spreading liquidity across more variants. We also discuss spot and forward pricing, batch versus live inference, and why an index can become disconnected from the cost of actually buying compute.
Independence is central to Evan’s case for a market operator. He argues that companies with large hardware or inference businesses have incentives that make it harder to run a neutral exchange. We also get into the unglamorous engineering: visiting sites, designing clusters, running the backplane, and making infrastructure programmable. Near the end, he describes the ability to subdivide InfiniBand fabric on the fly as one example.
Guest: Evan Conrad, CEO of SF Compute.
Show notes
Timestamps
0:00 — Why can’t NVIDIA build this?
0:25 — Welcome back to Latent Space
2:23 — From selling a room to running the hotel
3:47 — Physical settlement: actually delivering compute
6:00 — Liquidity and the GPU order book
8:42 — Trading nonstandard compute
10:00 — Interrupting and moving workloads
12:20 — Why hardware cohorts matter
14:32 — How far out do compute contracts go?
16:19 — Batch versus live inference leases
17:58 — The shape of compute prices
20:33 — GPU generations, specifications and SKUs
23:30 — Physical versus cash settlement
27:00 — The emerging compute market ecosystem
29:03 — What it takes to deliver a GPU cluster
32:24 — Sovereign compute
33:26 — Why NVIDIA can’t build this market
34:48 — A programmable data center
Transcript
Automated transcript, lightly formatted; speaker labels have not been verified.
00:00 NVIDIA is obviously like the 800-pound gorilla in the room. Why didn’t they make SF Compute? So our impression is that NVIDIA can’t make it, but benefits tremendously from it existing. The reason why they can’t make it is because they would then not be independent enough for people to trade on it.
00:25 Anyway, Evan, welcome back to Latent Space. Thank you so Welcome back to AIE. Actually, first time at AIE. I think so, yeah. This is a very special one because I feel like you have been working on compute trading, compute markets for a while. Now it’s like feeling of the moment. Yeah. Jump basically announced that they are getting into compute markets, which is why I’m excited to sort of cluster people. You met Oliver from SemiAnalysis, who is basically initiating coverage on compute markets is my read on this. So people are going to watch your talk.
01:00 I think it’s a very good explanation of both your origin story and why you need to exist and all the hard problems you work on. Yeah. Done in a very simple, engaging manner. I’m very jealous of your writing skills because you showed the flyer that you showed at New Reps. Yeah. I was like, it’s pretty straightforward and simple and it’s good. Yeah. So anyway, congrats. Thank you. I appreciate that. And also congrats on AIE. It seems to, y’all are rolling. We’re rolling. I just came back from the site tour for next year’s AIE. then for You have like a hint? At Javits Center. It’s the same place they do Comic Con. Okay. Wow.
01:35 Wow. Wow. Damn. But I think the finance focus is really hitting for folks. Yeah. So we’re going to try to split out. Like this is, here we just lump everything in finance. Like your finance, their finance, like insurance is finance, but like they’re all super different. Yeah. So we’ll probably break out things more minutely in future because people want to go deep on hedge fund stuff or banking or personal wealth. Yeah. Especially in New York. Yeah. Anthropic was like, yeah, we’re all about RIAs right now. Yeah. RIAs like registered investment advisors. Yes. Like so different from you.
02:10 No, no, no. I’m aware of this. Yeah. Yeah. Okay. So for listeners who don’t know, we did a podcast two years ago where you talked about your, you know, where SF Compute was. Um, today you’re, you’re much more in a sort of physical settlement. You’re talking, you’re talking about your, your back fabric or what do call it? Uh, the backplane. Backplane. Yeah. Which obviously that takes a lot of hardcore engineering stuff. Yeah. Um, can you give a status update on like, where is SF Compute versus listeners who last heard you on a podcast two years ago? So, uh, when we started, we were this company called Junark and then we were just trying to sublease our cluster.
02:45 Um, and we started there in the technical stack. Um, and a way to think about it is it’s like you’re in a hotel and you’re trying to sell that hotel room. And then somebody asks you, Hey, like, are you going to clean my room? Um, and then you say, Oh, I guess I probably also need the hotel room as well. Or like I need the, um, the cleaning closet. And so you go get access to the cleaning closet and then they want room service. And so you get access to the kitchen and then, you know, they want, um, security. And so you get access to the security closet and you get this and you do that and you do this. And at a certain point you’re just running the whole hotel. Um, and the moment you’re running the whole hotel, you say, Oh, wait a second.
03:21 Um, instead my supplier on this market, um, is somebody that looks closer to a, fund, um, or a cloud that has access in our case, a cloud that has access to the data center. And we’re going to level up. And so what we’ve spent the last several years doing is building the entire operating capacity of a cloud, um, in order to offer, in order to actually deliver, uh, the compute at a reliable level. Um, we call that physical settlement. We’re taking a term from the futures market, from futures markets, from other markets, um, where in, you know,
03:57 oil or soybeans or something, physical settlement means you get delivered the oil and the soybeans. Yes. Um, but, um, in some markets, um, like in oil and also in power, um, there’s like a delivery network in oil. You have to have pipelines. If you would like the oil to be delivered to you, there has to be a way for it to get to you. Yes. Um, and finance, and for people interested in financial history during COVID oil went negative because, uh, so like I will pay you to take oil from me because it is sitting there in the port. Uh, and like, I can’t pay for this.
04:32 Yes. And so I will pay you to take my oil. Yes. That is like the weirdness of physical settlement. Yes. There’s a bunch of weirdnesses like this in other markets as well. So in the power markets and the grid, um, there’s parts of the orders that you fill out, the buy orders and sellers that, um, are there because the boiler takes a while to cool off. Um, and you can, you know, stop the boiler and then it will, you can start it back up again within some period of time. And so that has to get factored into the order book. Um, this genre thing is the same thing that happens in compute markets. So we co-designed physical settlement and the order book at the
05:10 same time. And that meant that we get to actually handle the real constraints of compute, which might include something like, oh, I’m okay with being interrupted, but only if there’s like storage on the thing that I can recover from really quickly. Um, very similar types of problems. Yeah. So, uh, people can refer to your talk and you’ve done a couple other podcasts already. Um, so I want to be like the sort of 201 level instead of the 101 level. Sure. Uh, and also because I, I am, I have been a derivative trader, so I can give people a bit more detail that you normally don’t get into. Um, so for example, well, how, how mature or deep or liquid is the compute
05:45 market today? Uh, not very mature or deep or liquid. Okay. so what we’ve been doing is first we had to, we had to go get a bunch of supply online in the first place, which just takes a while. And then we had to build the entire platform in order to do this. Um, and we just now got that working. And so now we move, um, capacity over onto the order book, um, which makes it liquid. Um, and so 5% of our total capacity is on the order book and we’re consistently moving more and more. And then we’re bringing on more capacity. That’s all on the order book. Um, the nice part is that, um, maybe I should have said the, the volume on the actual order book is small, but the,
06:21 um, the liquidity on the order book is high. So for that small amount of capacity that you have, um, you have, um, like very concentrated liquidity. So, um, a sort of standard metric, I would say for someone who is using this is a customer who buys capacity from us and they spend a hundred dollars to buy that capacity. We’ll make back $25 on average from selling it back to somebody else. Um, and that’s the number that we, or this is one of the metrics that we try to track because, we want the liquidity to be useful to the commercial participant and you can measure liquidity in a bunch of different ways. You can say like, I just want tons of makers or something.
06:56 Um, and you can measure it in like high-minded ways. We try to measure it by like, is this solving the problem, um, for the customer? Yeah. Got it. And so, uh, last time we talked as well, there was a lot of contract, uh, conversation about standardizing the contracts. Yes. talking with like, uh, the CFTC or whatever, what’s the status of those things? Um, so I think standardizing the contracts in general is still pretty far away. Um, and the, what we were trying to do at the time was, um, or basically what we learned over the last several years was we just needed to
07:31 operate the whole thing. Otherwise you cannot standardize the thing. And so to get closer to something that can be standardized, you first need to make all the operations of the thing the same. Right. To give an example of what can be different is a single operator can offer two different types of virtual machines. One type of virtual machine, um, might have a 40% performance, um, degradation versus another types of virtual machine. And 40% is a lot. When you say performance degradation, are you talking like MFU type things or like amount of burn in that you’ve experienced or what? It will affect. So in this particular case, I’m talking about GPU kernel launches and how long it takes for that to happen.
08:07 Um, which is a process that happens quite frequently when you’re doing sort of either training or inference. Um, the, um, uh, that will basically, the end result is that you get a worse MFU, but, um, like is the intermediate level. the intermediate level basically. Yeah. Yeah. So, so what I’m trying to, so for anyone who’s traded commodities before, there’s different grades of commodity. Yes. Right. There’s like, it’s not oil. It’s light sweet crude versus like whatever the opposite of light sweet crude is, or it’s like, you know, uh, corn level a or corn B or whatever. Uh, and so like people are trying to, well, you have to establish the equivalents of those things. Correct. Yes. I’m wondering if they have been established.
08:42 So what we did instead, um, was we created an order book that had multiple different constraints and you can say, I want to have, um, you know, something that’s between this and that. Um, it runs from this time to this time. Um, it’s like, uh, X to Y, it’s a bunch of different properties that you can attach to your order. And then we match that against sell orders that have those properties, um, in a big matching engine. Um, and the reason why we did it this way is it allows us to generate liquidity without having to standardize before, know, what the technology can do. Um, and because the technology was shifting so quickly, um, we wanted to basically be able to do that. The trade off of this is that you have less concentrated liquidity.
09:18 And so the prices are going to have, there’s going to be more spreads between the, the buy and the sell. Yeah. Um, it sucks. Yeah. But on the other hand, it means there’s lots of arbitrage to make, um, if you can do that correctly. So if you have a product that can use lots of different types of capacity, um, then you might really like SF compute because you will be able to use lots of different types of capacity and you’ll make more money. Um, you can take the spread. Okay. What dimensions matter in this grid of things? Um, so a bunch of different things. Um, one is dimensions categories that I will say is like, is there a
09:53 workload running on it right now or is it capacity in the future? Sure. is it capacity right now? Um, capacity right now in order for you to sell that, um, you have to interrupt your current workload. So you have to think, do I care if my current workload is interrupted or not? Capacity that’s sold that’s starting a month from now, um, you can buy and sell that all the time. And so you get, um, liquidity on that really easily. Um, there is some concern like, oh no, well I have capacity next month, but it’s very different than if you’re interrupting a workload right now. Um, the way that the interruption happens matters. So an example is if you’re on a cluster and the cluster doesn’t have as amazing storage as you want it to have. Um, so an example is like, do you have a Weco or Vast system under the hood?
10:31 Um, and do you offer that? And like, what’s the size of that? Um, this changes how you respond to, um, being preempted or removed off the cluster. meanwhile, the location of that storage matters. So, um, if you’re able to move that data from one cluster to another, um, quickly, then maybe you might be able to spread across multiple different clusters. Um, and you can be interrupted over here and then pop back up over there. Um, to make that work, you also need, like a, a fast transit path between one cluster and another. Typically this is implemented with a WAN, but most of