# How to Maximize Memory Efficiency and Reduce Costs on NVIDIA Jetson

## Executive summary

This technical deep dive addresses the critical challenge of deploying large, complex AI models (LLMs and VLMs) on resource-constrained edge devices like the NVIDIA Jetson. Speakers demonstrate that significant memory optimization—achieving cost savings and increased model capability—is possible by implementing a combination of techniques: model quantization (e.g., INT4), optimizing the KV cache, utilizing efficient inference runtimes (like Llama.cpp), and optimizing the full system architecture. These methods allow models that initially require over 100 GB of memory to run on Jetson Orin NX SKUs with minimal performance degradation.

## Key takeaways

- Memory Optimization Goals: Optimization serves two main goals: cost reduction (allowing deployment on lower-cost Jetson SKUs) and increasing model intelligence (fitting larger, more capable AI models into existing hardware).
- Full-Stack Optimization Example: A complex robotics application (based on NVIDIA's Remember project) was optimized from requiring ~60 GB of memory on an AGX Orin 64 to fitting within 12.5 GB on a Jetson Orin NX 16, representing a 79% memory reduction. This enables running the entire stack on the robot without cloud dependency.
- VLM/LLM Memory Reduction: A large Vision-Language Model (VLM) was optimized from requiring over 100 GB of memory allocation to just 7.5 GB on an Orin NX, demonstrating massive memory efficiency gains through multiple optimization levers.
- Automated Optimization Workflow: The NVIDIA Jetson Agent Skills automate the memory optimization process in four steps: 1) Audit and benchmark the model, 2) Tune optimization levers (e.g., quantization, KV cache), 3) Benchmark again, and 4) Validate on the real use case.

## Technical details

- Memory Allocation Sources: For modern AI models, memory consumption is derived from three main sources: 1) Model Weights (the core model parameters); 2) Context/KV Cache Memory (stores tokens from previous requests, growing with context size); and 3) Activations/Input Processing (especially large for video/image encoders in VLMs).
- Optimization Levers: Four primary levers for memory optimization are: 1) Weight Optimization (e.g., moving from FP16 to INT8 or INT4); 2) KV Cache Optimization (reducing context size or adjusting precision); 3) Runtime Optimization (reducing batch size or simultaneous requests); and 4) Model Specific Optimizations (reducing image/video resolution).
- Quantization and Frameworks: The use of Quantization Aware Distillation (QAD) is recommended for VLMs/LLMs to minimize the loss of accuracy when reducing model precision. Efficient runtimes like Llama.cpp are highlighted for their flexibility in memory management compared to frameworks like vLLM or SGLang, especially on Ampere-based platforms.
- Memory Handling on Jetson: Jetson platforms use unified memory, meaning process memory and GPU memory (NVMap) are shared. Performance depends on ensuring that the total memory usage over time does not exceed the physical limit.

## Practical implications

- Build engineers can significantly reduce hardware costs by targeting lower-cost Jetson SKUs (e.g., Orin NX 16) instead of high-end, memory-intensive platforms (e.g., AGX Orin 64).
- The optimization process is systematic and can be automated using NVIDIA Jetson Agent Skills, reducing manual effort and time-to-deployment.
- System design must account for the trade-offs between memory reduction (e.g., lower precision) and potential degradation in model accuracy or reasoning capability.
- Full-stack optimization is crucial; optimizing the model is not enough—the entire robotic architecture must be streamlined.

## Topics

Edge AI, Memory Optimization, Large Language Models (LLMs), Vision-Language Models (VLMs), Embedded Systems, Quantization, NVIDIA Jetson Agent Skills, TensorRT, Llama.cpp, Connect Tech/Marco's Article

Source: https://www.youtube.com/watch?v=mW1LWZWkojo