Nvidia says Groq racks will be online this year following $20 billion purchase
Nvidia's race to manufacture Groq chips and make them available to customers highlight the growing importance in AI of low-latency inference.
TL;DR
- Nvidia's Groq 3 LPX rack is in full production.
- The Groq rack technology comes from Nvidia's $20 billion acquisition of chip startup Groq.
- The Groq rack will be deployed at neocloud Nebius later this year.
- Low-latency inference is crucial for responsive AI agents, especially for coding.
- Cloud companies can charge more for services utilizing low-latency chips.
- The Groq architecture features 500MB of SRAM on-chip to reduce memory bottlenecks.
- Nvidia's Groq 3 LPX rack can deliver 3,400 tokens per second.
- This technology competes with other low-latency inference solutions from companies like AMD and Cerebras (powering OpenAI's Ultrafast mode).
- Low-latency chips focus on the 'decode' phase of serving models and do not replace GPUs.
- Nvidia CEO Jensen Huang projected $1 trillion in cumulative sales for Blackwell chips and Vera Rubin systems through 2027.
- Huang plans to allocate a quarter of data center space for coding applications to Groq chips.