AI Scale-Up and Scale-Out Interconnects

Abstract:

The rapid progress of modern AI datacenters is increasingly limited by the ability to efficiently interconnect large numbers of compute devices. As model size, training scale, and inference complexity continue to grow - interconnect bandwidth, latency, reliability, power efficiency, and cost have emerged as first-order constraints shaping AI datacenter architecture. The rapidly evolving AI infrastructure must therefore balance two distinct communication regimes: scale-up interconnect, which tightly couple accelerators to behave as a single logical device, and scale-out interconnect, which connects many such domains across racks and clusters to enable distributed training and inference at datacenter scale.

This talk examines the architectural distinctions between scale-up and scale-out, comparing emerging protocol-layer options such as NVLink, UALink, and Ethernet-based approaches, alongside physical-layer trade-offs spanning copper, pluggable optics, co-packaged optics (CPO), near-package optics (NPO), and emerging alternatives including micro-LED and RF interconnects. We discuss the challenges of scaling copper beyond ~200 Gb/s per lane, the trade-offs introduced by optical and alternative solutions for scale-up, the need for high-radix switching in scale-out networks - and conclude with how these challenges are driving a large and durable market opportunity for startups operating across silicon, packaging, connectors, cables, and system architecture.

Bio:

Manish Muthal is a Sr. Managing Director at Maverick Silicon and serves on the boards of Axiado, Baya Systems, GSME, and SMART USA Institute. Prior to joining Maverick, Manish was a Senior Vice President for Hyperscaler engagements at KIOXIA. He has served in executive roles at semiconductor companies such as Intel, Xilinx, Broadcom, and LSI, where he built and led organizations that delivered record revenue growth. He has also co-founded multiple venture backed start-ups, and has been a corporate investor, sponsor, and board advisor at multiple successful companies at their earliest stages.

He received his B.S. from Visvesvaraya National Institute of Technology Nagpur, India, and his M.S. in Computer Engineering from the University of California Santa Barbara.