# Building real-time vision & edge navigation with Gemini Live

## Executive summary

This video details the creation of Sprout, a semi-autonomous plant robot designed to navigate indoor environments, locate sunlight, and monitor the health of other plants. The project successfully integrated a TurtleBot 4 base with advanced AI capabilities, leveraging the Gemini ecosystem (including Gemini Flash for vision and ER 2.0 for segmentation) to achieve real-time edge navigation and interactive voice communication via Gemini Live.

## Key takeaways

- Autonomous Navigation and Sensing: The robot uses a TurtleBot 4 base, equipped with a camera and LiDAR, allowing it to map rooms, navigate to find light sources, and detect obstacles. It also features a planter sensor to monitor if the plant is thirsty.
- AI-Powered Vision and Interaction: The system utilizes Gemini Flash for vision tasks and ER 2.0 for segmentation. Gemini Live provides the voice interface, allowing the robot to communicate and determine actions autonomously in specific scenarios.
- Rapid Prototyping Cycle: The speaker highlights the speed of development, noting that the entire software build was completed in approximately two weeks using Gemini as a core development copilot.

## Technical details

- Hardware Stack: The physical platform is built on a TurtleBot 4 base (described as a Roomba chassis with camera and LiDAR) and features a Pixel 11 for the visual components.
- AI/Software Stack: The software architecture relies on Gemini, specifically utilizing Gemini Flash for vision processing, ER 2.0 for image segmentation, and Gemini Live for voice interaction. Nano Banana was used for sketching the casing design.
- Development Framework: The software was built using Gemini in Antigravity.

## Practical implications

- The architecture demonstrates a rapid, end-to-end approach to building semi-autonomous edge devices using large language models (LLMs) and specialized vision APIs.
- The combination of mapping (LiDAR), object detection (Gemini Flash), and environmental sensing (planter sensor) suggests applicability for general indoor maintenance or monitoring robots.
- The use of Gemini Live for voice interaction allows the robot to exhibit complex, conversational behavior.

## Topics

Robotics, Edge AI, Computer Vision, LLMs, Build Engineering, Gemini, Google AI Studio, https://goo.gle/4BmThG6

Source: https://www.youtube.com/watch?v=ID6vEy94j7c