Silicon Co-Design
An Advanced System Architecture Breakdown of OpenAI’s Jalapeno Inference Accelerator
Begin with the end goal in mind: how user experience defined the architecture and how agentic coding methods contributed to the shortened chip design timeline
At the end of the Hot Chips 2026 conference, OpenAI presented their chip architecture, Jalapeno. Jalapeno is a general purpose AI inference accelerator designed primarily for OpenAI’s workloads but generalizable to frontier model workloads.
Most commentators will look at the headline and think OpenAI built a magic black box that replaces GPUs overnight. This couldn’t be further from the truth. There is a rich set of system architecture tradeoffs and design methodology improvements that went into the chip design.
This post is an advanced case study into Jalapeno based on their presentation deck and related material. I will cover the entire architecture, including what motivated it, what tradeoffs and alternatives were likely considered by their leadership , and how AI added value:
Part 1: System Specification and Constraints
The Pareto Frontier - System Level Tradeoffs Latency of LLM Requests
Inference Efficiency - Energy per Request
Architectural Observations Theoretical Memory Roofline is Higher than User Token Throughput
Unpredictable Workload Characteristics between Prefill, Speculate, and Decode
Balancing Large KV States on a Single Chip
Sources of Latency that affect User Experience Network Interconnect with SOTA Breakdown
Compute Stalls Caused by Operands Arriving Late
Part 2: How the Hardware Architecture was Co-designed to Handle LLM Workloads
Network Architecture with Broadcom’s Tomahawk 6 Switch Alternative Architectures (2-Tier Fat-Tree, 3D/4D Direct Torus / Mesh Network, and Dragonfly)
Speculative Decoding with Multi-Token Prediction
Part 3: How AI Assisted Chip Design
🔒Three Major Agentic Coding Paradigms to Assist in Chip Design 🔒LLM - RTL Pipeline
🔒LLM - High Level Synthesis Pipeline
🔒Why Agentic Flows Aren’t in Leading-Edge Chip Design Yet
🔒How Jalapeno used AI: High Level Synthesis with Google XLS 🔒Compared to “Waterfall” Chip Design Methodology
Part 4: SW Model Tuning and Performance Results
After writing this post, I wonder whether AI should be best tailored to enhance the steps within the current “waterfall” chip design methodology or enable a more iterative methodology. Throughout this design, OpenAI leveraged an open source tool Google XLS that aided in block level optimization for PPA and formal verification leveraging their frontier models. Google XLS had its benefits in contributing to a shorter timeline, but I think the iterative, cross-domain (and often messier) nature of work is perhaps the more significant factor.
Prior to Hot Chips, I studied the architecture of data centers from the ground up by attended five technical conferences. You will see several of my deep dives scattered throughout this post that form the basis of various cross-domain effects discussed here.
Bridging the Silos of the AI Data Center: My Journey Through Five Major Technical Conferences (ISSCC, DesignCon, APEC, ECTC, and DAC)—and Advice for Engineers
After DAC, I wrote a deep dive on how AI is currently employed in Chip Design. I recommend you read this and a primer on the fundamentals of AI accelerators.
Design Automation Conference 2026 - The Bottlenecks that Limit AI Use in Chip Design
A Comprehensive Overview of the AI Accelerator Landscape: From Digital to Analog Compute-in-Memory Architectures
As always, if you’re an expert and notice a mistake in this article, please reach out to me so I can have it promptly corrected.
Silicon Co-Design is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.
Part 1: System Specification and Constraints
Of all the chip architectures presented at Hot Chips, Jalapeno captures a lot of complex tradeoffs facing the system definition and co-design of AI accelerators. Rather than optimizing for peak “show” FLOPs, Jalapeno was designed with end-to-end optimization in mind while maintaining balanced performance across frontier model workloads.
The Pareto Frontier - System Level Tradeoffs
Architecting custom silicon requires navigating through tightly coupled Pareto frontiers. Pareto frontiers across three macro-level layers include:
Intelligence vs Service Cost vs Latency. At the API / model layer, more intelligent models utilizing higher parameters counts, reasoning trees, or MoE improve performance, but increase memory footprint and FLOP requirement per token.
Throughput vs Interactivity. At the systems / inference layer, there is a tradeoff between prefill (compute heavy) and decode (memory heavy) operations where batch sizes have an impact.
Compute Density (FLOPs/mm2) vs Energy per Bit vs Reticle Limits . At the physical silicon and packaging level, increasing on-chip compute density is bound by lithography reticle limits. This forces a shift to chiplets to separate specialized functions in appropriate process nodes and 2D interconnects to connect them.
In additional, there are a few other important system-level and microarchitectural pareto frontiers:
Memory BW vs Capacity vs Energy per Bit. This is the classic “Memory wall” tradeoff where on-chip SRAM has massive BW but terrible area density. HBM has huge capacity, but limited by thermals and shoreline limits. You need a mix of both.
Switch Radix vs Reach. In the scale-up interconnect, high-radix switches (such as Broadcom TH6) better “centralize” multiple GPUs by reducing network hops. However, a high switch radix is only as good as the max reach the furthest GPU can connect to it. Thus, copper SI limits and optical power overhead often bounds the max number of GPUs that can reasonably connect to high radix switches in the given network architecture.
Programmability vs Silicon Efficiency. In general, more efficient silicon for a specific task tends to be less programmable. GPUs are highly programmable while fixed function, highly efficient ASICs are less so.
Model Accuracy vs Compute/Memory Density. Most AI Accelerators push the precision down to FP8 and FP4 to increase the compute density and ease memory BW pressure. However, this decreases dynamic range and add more compiler complexity to ensure model accuracy is maintained.
Basically, the most powerful model in the world can define what is possible, but needs effective HW-SW co-design to deploy it economically.
OpenAI considered these pareto frontiers and narrowed it down to two important ones:
End-to-End Request Latency vs Mixed Throughput / kW.
Decode Speeds vs Mixed Throughput / kW.
Throughout this post, I will touch upon design considerations in several other pareto frontiers as well.
Latency of LLM Requests
Traditional semiconductor vendors tend to architect accelerators by optimizing HW metrics such as peak theoretical FLOPs (or “show FLOPS”) and unconstrained batch throughput.
OpenAI inverted this paradigm and began with what the end user cared about. OpenAI observed that user experience is largely covered by inference latency that is typically quantified in three ways:
Time to First Token (TTFT) - Prompt processing speed during the prefill phase
Time per Output Token (TPOT) - The speed of token generation during the decode phase
Time to Last Token - Total turnaround time of the whole request
Starting from the end-user requirements of ChatGPT, real-time API clients, and internal agentic workflows, OpenAI reasoned backwards to establish strict SLAs—specifically targeting TPOT and end-to-end request latency - all while maintaining generality to other frontier models.
Inference Efficiency - Energy per request
At the cluster level, inference efficiency effectively sets a constraint on the power system of the rack itself for the aggregate users expected to use it.
Inference energy competes with energy per request based on the batch size :
Smaller batches enable lower single user latency across fast interconnects, but wastes idle compute logic because memory channels stream 10’s of GBs of weights per token pass.
Larger batches maximizes energy efficiency to keep ALUs operating, but introduces queueing delays for single users.
Both tradeoffs much be balanced to enhance overall user experience while ensuring economic feasibility and grid usage. Depending on the service tier, OpenAI defines an end-to-end latency target through a service level agreement (SLA). After this is satisfied, compute efficiency is optimized (requests / s / W). Basically:
Architectural Observations
OpenAI made a few architectural observations about comparable GPUs that influenced their chip and system architecture.
Theoretical Memory Roofline is Higher Than User Token Throughput
OpenAI’s calculation of the theoretical memory roofline highlights a massive gap between the theoretical memory BW and actual token generation speeds.
OpenAI calculates a theoretical “max” token throughput of ~2000 full model weight reads / second IF HBM throughput was the solve physical constraint. They assume a 128 chip aggregate HBM4 BW of 1 PB/s with a ~1T parameter frontier model with FP4 values. This max limit can be pushed up to 5000 - 10000 tokens / s per user with speculative decoding.
In reality, single user token rates hover around 20 - 200 tokens / second. This is limited by various overheads in parallelism, such as inter-chip network latency, synchronization delays, and NoC routing contention. This overhead often causes underutilization of HBMs that sit idle between tokens.
To recover cluster economics, operators typically increase the batch size of concurrent requests to saturate available HBM bandwidth. While batching optimizes server throughput, it does nothing to speed up the individual user prompts. Users are completely unaware and often don’t care about the other requests that are batched alongside theirs.
This memory BW can be pushed even higher, but runs into severe physics constraints I fully describe in this article, causing memory makers like SK Hynix to look into optical interconnects.
A Comprehensive Deep Dive into High-Speed HBM5 Interconnect: Signal Integrity and Co-Design Challenges
Unpredictable Workload Characteristics between Prefill, Speculate, and Decode
Virtually all AI accelerators and surrounding systems face a tradeoff between generality and specificity to successfully handle varying degrees of workloads needed to run inference on an LLM request. It would be optimal to have accelerators fully utilized 100% of the time, but this is hard to achieve in practice.
The figure above shows the major LLM processing steps: prefill and decode. In between these is an optional third step: speculate.
Prefill constructs the attention context by taking in the input prompt tokens, passes them all in parallel through a transformer model and and populating the key-value (KV) Cache in memory. Prefill is compute heavy where the time-to-first-token (TTFT) matters.
Prefill stresses the compute that affects the PDN di/dt and thermals where throughput / kW matter
Prefill also stresses local SRAM and interconnect BW before committing data to memory
APEC 2026: A Complete Overview of the AI Data Center Power System
Advanced Packaging Co-Design: The Thermodynamic and Mechanical Constraints of High Power GPUs
A Deep Dive into SRAM: The Staging Ground of LLM Inference
Speculate is an optional step that runs a lightweight “draft” model to predict the next K (say 5-10) tokens. These K forward passes are fairly cheap and are ran a single verification forward pass across all candidate tokens simultaneously to verify results. Speculate is primarily used to bypass the memory BW wall of the decode step
Speculate can take the form of single-token or multi-token decode
Speculate stresses the optical / SerDes system because communication latency matters when data packets traverses stages.
Pushing the Speed Limit: Designing SerDes Transceivers for the 224 and 448Gbps Scaling Era
Decode autoregressively generates the response to the request one token at a time. To generate each output token, the model must read the model weight + accumulated KVcache to calculate attention and compute a single token. Decode is largely a memory bound operation that stresses the HBM because it requires streaming fill target model weight with All-to-All (in MoE) for every pass.
Decode can be optimized with techniques such as Grouped-Query Attention (GQA) and speculative decoding that help circumvent memory BW
Decode also tends to make workloads bursty in MoE models
The problem is that most workloads ratios vary wildly, making it difficult it is to perfectly anticipate the compute needs for every request.
You can perhaps design highly specialized chip to handle each type of workloads and chain them all up. However, this could lead to compute inefficiency from idle accelerators.
Likewise, you can implement all three computations on a single balanced chip where parts of it are potentially underutilized.
Ravi Narayanaswami observed that it’s worse to have future unsupported features than built in underutilized features. He mentioned how “opportunity cost is higher than marginal cost because it has a greater regret factor”. This means its worth the additional marginal cost up front to include features than may be underutilized in “dark silicon” in the short term.
My take on this is that its almost like how we don’t design rigid, highly efficient CPUs to handle fixed options to meet our needs now; we need to allow for additional programmability on general enough chips for now to handle workloads in the future. When designing new AI accelerators, I don’t think its sufficient to look at raw performance on current workloads for show; you have to consider the longer term uncertainty as well.
Balancing Large KV States on a Single Chip
The size of HBM and LLM models imposes several challenges on KVCache to handle large context windows from increasingly large files passed on by users (10-100’s GB / request).
One approach is to build specialized chips for each type of workload. However, this entire KV cache still needs to be transported over the entire external network.
OpenAI does not believe that this is the long term solution. It believes that dark silicon is a lesser evil than idle specialized accelerators . Separate accelerators add to additional HW complexity that takes up space for other useful functions. Inactive blocks on a single balanced chip silicon minimizes complexity and black boxes chip functionality to users.
OpenAI implements a single chip solution with KVCache in close proximity to compute, memory, and the network. This way, data movement is balanced across all three components.
Sources of Latency That affect User Experience
OpenAI is trying to cut down is latency, or more specifically, tail latency, to enhance user experience .
In LLM inference, data is primarily transported through global reduce operations such as all-reduce and reduce-scatter. These operations transmit data across multiple processors to keep parallel data notes in sync.
Tail latency cause a straggler effect across millions of computations where the worst case data packet can hold up a big computation where all other data packets arrive in time.
OpenAI notes a few sources of latency within the signal path and at the SW level:
Network Latency - the signal path from point A to point B incurs various latency limits due to time-of-flight and HW complexity.
Memory Systems - two things affect memory latency: HBM read/writes have inherent latency that typically ranges from 30 - 50 nanoseconds due to signal traversal in the TSV stack, reading the DRAM cell itself, and complexity in the base die.
Unified, aggregated memory systems tend to have highly contending paths due to NoC routing, crossbar intersections, and bank conflicts
Long Latency Path in Interconnect
In general, any additional circuit complexity to handle high speed signal integrity nonidealities adds latency.
Signals that travel through scale up/out network interconnects flow through switches that each add latency. Shown above are some ballpark SOTA numbers as of Feb 2026, and we see that the time-of-flight (or what OpenAI referred to as “HW speed of light”) is small relative to other sources.
High-Speed Signal Integrity: Physical Impairments and Equalization Architectures
Several things can be done to minimize latency from all possible sources. Nikola Nedovic points out a few things during an ISSCC 2026 forum “Emerging Low-Latency Optical Connectivity”:
Network Architecture level - Increase the switch radix to decrease the # of network “hops” that packets take
Rack Level - Connect GPUs directly with massive copper backplanes rather than traditional PCIe controllers, retimers, and ethernet / InfiniBand switches
Component level - Optimize FEC that introduces 10-100’s ns of fixed delay
Chip Level - Move to low latency CPO links. This is the most widely talked approach attracting a lot of investment attention. NVIDIA is pursuing WDM to scale throughout without increasing data rate, and Marvell is targeting OMIB with GeSi modulators. A Complete Deep Dive of NVIDIA DWDM Co-Packaged Optics for Scaling beyond 200Gbps Chad · Jun 23 Read full story A Deep Dive of GeSi Electro-Absorption Modulators and Marvell/Celestial AI's OMIB for 200Gbps+ Interconnects Chad · Jul 8 Read full story
In my opinion, OpenAI did not utilize CPO because it is not really ready for real time deployment and looked to other means to optimize latency in the present term.
Compute Stalls Caused by Operands arriving late
Accumulated latencies in the network from these physical sources cause operands to arrive late in registers. In the classical computer architecture sense, operands arriving late leads to “stalling” because PE arrays require precise lockstep synchronization of data. This leads to compute units sitting idle that are waiting for results of intermediate operations.
Latencies accumulate from all of the physical effects in the previous section. They also accumulate from network congestion and contention where two data packets try to transmit on the same physical resource, such as reading / writing from the same memory bank or output node. These are some reasons why many consider the interconnect one of the critical bottlenecks of scaling AI compute, both from a fundamental physics and networking standpoint.
Silicon Co-Design is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.
Part 2: How the Hardware Architecture was Co-designed to Handle LLM Workloads
These pareto tradeoffs and architecture observations influence the HW system architecture. OpenAI used Broadcom’s Tomahawk 6 switch and Celestica for rack scale integration.
A System-Level Overview of Scale-up AI Infrastructure Racks (Part 1: Network Architecture)
Network Architecture with Broadcom’s Tomahawk 6 Switch
OpenAI uses a “half flattened” bounded 2-hop Clos topology for predictable latency. This