# Accelerate Visual AI Agents on NVIDIA Jetson with VSS 3.3 and Cosmos

## Executive summary

This livestream demonstrates how to accelerate the development and deployment of visual AI agents on the NVIDIA Jetson edge platform using NVIDIA Blueprint for Video Search and Summarization (VSS) 3.3 and Cosmos. The workflow leverages coding agents and specialized skills (Jetson Device Skills, BSP Skills) to transform natural-language requests into configured, scalable AI applications, enabling complex tasks like multi-camera fusion search and automated assembly inspection.

## Key takeaways

- Agentic Development Workflow on Jetson: A developer defines an application via a simple prompt to a coding agent (e.g., Open Claw, Codex). The agent then accesses specialized skills (Jetson, VSS, DeepStream) to build and deploy the multi-model pipeline, significantly expediting a process that might take weeks into a matter of hours.
- VSS as a Scalable Blueprint: VSS acts as a vision agent blueprint that connects live camera feeds, recorded video, and user requests into a scalable agent capable of Observe, Reason, and Act. It provides both 'always-on' tools (real-time analysis, tracking) and 'on-demand' tools (search, summarization, report generation).
- Multi-Model Fusion and Efficiency: The system achieves efficiency by using lighter-weight models (like DDMnet or D1) as fast, always-on screeners to identify candidate events, which are then passed to resource-intensive VLMs (like Cosmos) for deeper, contextual reasoning, thus reducing VLM processing load and freeing up GPU capacity.
- Model Agnosticism and Deployment: VSS is designed to be service agnostic. Developers can connect their own VLMs or embedding models, and the system supports running components locally on the Jetson (e.g., Jetson Nano 8 GB) or pointing them to remote cloud endpoints.

## Technical details

- SOP/Assembly Procedure Checking: This workflow uses DDMnet (Dense Difference Maps) to examine visual feature changes over time, identifying activity boundaries. These segments are then processed by a fine-tuned NVIDIA Cosmos model, which identifies the specific assembly action (e.g., 'installing a fan'). The SOP checker compares the recognized sequence against a predefined checklist to flag missing steps.
- Traffic Monitoring Pipeline: The traffic monitoring demo uses a Decision Model (e.g., Liquid AI's D1 2B) as a fast, always-on screener. D1 scores the monitoring question ('Is there a queue?') every two seconds. Only if the 'Yes' score reaches a threshold (e.g., 85%) for three consecutive checks is a candidate event created for NVIDIA Cosmos to perform detailed verification.
- Resource Constraints and Scaling: The number of camera streams a Jetson can handle depends heavily on the resolution, frame rate, and codec. For example, 1080p at 60 FPS is estimated to handle around 24 streams. VSS itself does not use VRAM; the required VRAM for Cosmos depends on the model's quantization, ranging from 2 GB (smallest quantization) up to 30-70 GB (super models).
- Security and Extensibility: VSS blueprints operate within an 'Open Shell' secure sandbox, which dictates the policy of what the agents can access (APIs, CLIs, MCPs). The framework is designed to allow easy swapping of different VLMs via coding agents or provided documentation.

## Practical implications

- Engineers can significantly reduce development time by using agent skills to automate the creation and configuration of complex, multi-model pipelines.
- The architecture allows for cost reduction by optimizing VLM usage; only critical clips are sent to the VLM for deep reasoning, rather than processing all video continuously.
- The system supports both local edge deployment (Jetson) and cloud integration, allowing services to scale as needed.
- The VSS blueprint provides a robust, secure starting point that can be customized and extended by developers.

## Topics

Visual AI, Edge Computing, Video Analytics, Agentic Systems, NVIDIA Jetson, VLM, VSS 3.3 Tech Blog, VSS github, Brev Launchable

Source: https://www.youtube.com/watch?v=c0gdviC_OeI
