Accelerate Visual AI Agents on NVIDIA Jetson with VSS 3.3 and Cosmos
Summary
This livestream demonstrates how to accelerate the development and deployment of visual AI agents on the NVIDIA Jetson edge platform using NVIDIA Blueprint for Video Search and Summarization (VSS) 3.3 and Cosmos. The workflow leverages coding agents and specialized skills (Jetson Device Skills, BSP Skills) to transform natural-language requests into configured, scalable AI applications, enabling complex tasks like multi-camera fusion search and automated assembly inspection.
Key takeaways
-
Agentic Development Workflow on Jetson
2:33
A developer defines an application via a simple prompt to a coding agent (e.g., Open Claw, Codex). The agent then accesses specialized skills (Jetson, VSS, DeepStream) to build and deploy the multi-model pipeline, significantly expediting a process that might take weeks into a matter of hours.
-
VSS as a Scalable Blueprint
6:28
VSS acts as a vision agent blueprint that connects live camera feeds, recorded video, and user requests into a scalable agent capable of Observe, Reason, and Act. It provides both 'always-on' tools (real-time analysis, tracking) and 'on-demand' tools (search, summarization, report generation).
-
Multi-Model Fusion and Efficiency
9:58
The system achieves efficiency by using lighter-weight models (like DDMnet or D1) as fast, always-on screeners to identify candidate events, which are then passed to resource-intensive VLMs (like Cosmos) for deeper, contextual reasoning, thus reducing VLM processing load and freeing up GPU capacity.
-
Model Agnosticism and Deployment
13:11
VSS is designed to be service agnostic. Developers can connect their own VLMs or embedding models, and the system supports running components locally on the Jetson (e.g., Jetson Nano 8 GB) or pointing them to remote cloud endpoints.
Technical details
-
SOP/Assembly Procedure Checking
863s
This workflow uses DDMnet (Dense Difference Maps) to examine visual feature changes over time, identifying activity boundaries. These segments are then processed by a fine-tuned NVIDIA Cosmos model, which identifies the specific assembly action (e.g., 'installing a fan'). The SOP checker compares the recognized sequence against a predefined checklist to flag missing steps.
-
Traffic Monitoring Pipeline
1198s
The traffic monitoring demo uses a Decision Model (e.g., Liquid AI's D1 2B) as a fast, always-on screener. D1 scores the monitoring question ('Is there a queue?') every two seconds. Only if the 'Yes' score reaches a threshold (e.g., 85%) for three consecutive checks is a candidate event created for NVIDIA Cosmos to perform detailed verification.
-
Resource Constraints and Scaling
1341s
The number of camera streams a Jetson can handle depends heavily on the resolution, frame rate, and codec. For example, 1080p at 60 FPS is estimated to handle around 24 streams. VSS itself does not use VRAM; the required VRAM for Cosmos depends on the model's quantization, ranging from 2 GB (smallest quantization) up to 30-70 GB (super models).
-
Security and Extensibility
1673s
VSS blueprints operate within an 'Open Shell' secure sandbox, which dictates the policy of what the agents can access (APIs, CLIs, MCPs). The framework is designed to allow easy swapping of different VLMs via coding agents or provided documentation.
Mentioned resources
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.