# Serving GLM-5.3 Across Two NVIDIA DGX Stations with vLLM

## Executive summary

This tutorial demonstrates how to serve a frontier-scale, 753-billion-parameter Mixture-of-Experts (MoE) model, GLM-5.3, across two NVIDIA DGX Stations. By utilizing vLLM and tensor parallelism, the model's weights are split across two GPUs, allowing the system to maintain a single, OpenAI-compatible API endpoint without CPU offload. The process emphasizes robust infrastructure setup, including RDMA network configuration and synchronized software deployment, culminating in performance benchmarking using NVIDIA AIPerf.

## Key takeaways

- Scaling Models with Tensor Parallelism: When a model's checkpoint (e.g., GLM-5.3 NVFP4 at 464 GB) exceeds the memory of a single GPU (e.g., 252 GB on a GB300), tensor parallelism is used to split the weights across multiple GPUs, enabling the model to run locally.
- Infrastructure Requirements: Successful deployment requires connecting the DGX Stations via a direct network rail (e.g., ConnectX-8) to establish an RDMA connection, ensuring GPU-to-GPU communication bypasses standard TCP sockets.
- Deployment Workflow: The process involves setting up passwordless SSH, pulling the identical vLLM container image on both stations, copying the full checkpoint to both, and launching the vLLM script with `tensor parallel size two` on both machines.

## Technical details

- Network Setup and Prerequisites: The setup requires identifying the RDMA device and noting the IP addresses for both stations. Key network requirements include a 400 Gigabit link, MTU 9000, and synchronized clocks. The system must also have Docker available on both stations.
- Model Serving Initialization: The vLLM script is launched on both stations, specifying `tensor parallel size two`. This configuration ensures that both GPUs compute every request, keeping all model weights in GPU memory. The maximum model length is 98,304 tokens. The system utilizes FP8 KV cache and multi-token prediction for efficiency.
- Verification and Testing: Verification steps include checking the API health (`Half okay`), confirming RDMA communication via NCCL logs, and verifying the worker status using `nvidia-smi` (showing `TP zero` and `TP four`). Performance is measured using NVIDIA AIPerf with SPEED-Bench Coding prompts.

## Practical implications

- Enables running frontier-scale models locally on private hardware, which is crucial for working with sensitive data.
- Allows for prototyping AI agents and iterating on prompts without incurring per-token API costs.
- Provides a blueprint for scaling large language models (LLMs) across multi-node GPU clusters using tensor parallelism.

## Topics

Large Language Models (LLMs), Tensor Parallelism, vLLM, RDMA, NVIDIA DGX, Model Quantization (NVFP4), Performance Benchmarking (NVIDIA AIPerf), Connect Two DGX Stations playbook, NVIDIA Technical Blog

Source: https://www.youtube.com/watch?v=opRXPRnSkxk
