NVIDIA Developer

Serving GLM-5.3 Across Two NVIDIA DGX Stations with vLLM

Published 2026-11-14 · Duration 6:41

Summary

This tutorial demonstrates how to serve a frontier-scale, 753-billion-parameter Mixture-of-Experts (MoE) model, GLM-5.3, across two NVIDIA DGX Stations. By utilizing vLLM and tensor parallelism, the model's weights are split across two GPUs, allowing the system to maintain a single, OpenAI-compatible API endpoint without CPU offload. The process emphasizes robust infrastructure setup, including RDMA network configuration and synchronized software deployment, culminating in performance benchmarking using NVIDIA AIPerf.

Download summary

Key takeaways

  1. Scaling Models with Tensor Parallelism

    When a model's checkpoint (e.g., GLM-5.3 NVFP4 at 464 GB) exceeds the memory of a single GPU (e.g., 252 GB on a GB300), tensor parallelism is used to split the weights across multiple GPUs, enabling the model to run locally.

  2. Infrastructure Requirements

    Successful deployment requires connecting the DGX Stations via a direct network rail (e.g., ConnectX-8) to establish an RDMA connection, ensuring GPU-to-GPU communication bypasses standard TCP sockets.

  3. Deployment Workflow

    The process involves setting up passwordless SSH, pulling the identical vLLM container image on both stations, copying the full checkpoint to both, and launching the vLLM script with `tensor parallel size two` on both machines.

More digests on Large Language Models (LLMs)

Technical details

  • Network Setup and Prerequisites 0s

    The setup requires identifying the RDMA device and noting the IP addresses for both stations. Key network requirements include a 400 Gigabit link, MTU 9000, and synchronized clocks. The system must also have Docker available on both stations.

  • Model Serving Initialization 0s

    The vLLM script is launched on both stations, specifying `tensor parallel size two`. This configuration ensures that both GPUs compute every request, keeping all model weights in GPU memory. The maximum model length is 98,304 tokens. The system utilizes FP8 KV cache and multi-token prediction for efficiency.

  • Verification and Testing 0s

    Verification steps include checking the API health (`Half okay`), confirming RDMA communication via NCCL logs, and verifying the worker status using `nvidia-smi` (showing `TP zero` and `TP four`). Performance is measured using NVIDIA AIPerf with SPEED-Bench Coding prompts.

Mentioned resources

Channel & topics

Watch on YouTube · Back to latest

This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.