OpenAI has released the first benchmark results for Jalapeño, its custom inference system developed with Broadcom. Tested on SemiAnalysis’ InferenceX benchmark, Jalapeño delivered more tokens per user and higher throughput per kilowatt than the state-of-the-art inference processors included in the comparison.
OpenAI shared the results at the Hot Chips conference on Tuesday as part of a more detailed look at Jalapeño. Richard Ho, OpenAI’s head of hardware, said the system can process more AI workloads per unit of power while also returning responses with lower latency.
Jalapeño Targets Higher Throughput and Lower Latency
The comparison included a system based on Nvidia’s Blackwell processors. However, Jalapeño is not expected to reach meaningful deployment until 2027, giving competing hardware additional time to advance.
Ho said Jalapeño is expected to begin deploying in very small volumes at the end of 2026. Larger-scale deployment is planned for 2027.
OpenAI first announced Jalapeño last October and developed the system in close collaboration with Broadcom. The company’s own AI models were also used during the development process.
OpenAI intends to develop Jalapeño as a multigenerational platform, with AI products, models, chips, and memory designed together. The company previously described the processor as its first custom inference chip developed with Broadcom.
OpenAI Designs Around Inference Bottlenecks
The integrated design allowed OpenAI to focus on specific stages of inference that can slow processing. Jalapeño is designed in particular to reduce delays during the prefill and communication phases.
OpenAI said it designed the system to limit unnecessary data movement and communication delays. Model state, including the KV cache used while generating responses, can be placed and kept locally while the system activates the necessary combination of compute, memory, and networking for each inference phase.
According to TechCrunch, Ho said Jalapeño can serve large numbers of users efficiently while also supporting low-latency responses.
Featured image credits: Wikimedia Commons
For more stories like it, click the +Follow button at the top of this page to follow us.