Hi,
We’re sharing our custom ML pipeline, which works on live video footage to identify (using bounding boxes), mask and classify unusual objects during the exploration task. The pipelines works as a ROS2 package, and requires camera data, LIDAR data (including the GPS service).
Here is the brief overview of the pipeline:
- Grounded SAM 2: The live video is fed to Grounded SAM2, which has the following functional components:
- Grounded DINO classifies objects using text prompts like “weird object” or “car” and creates bounding boxes around them.
- SAM 2 creates masks for the object in the bounding box, thus separating the object from the background. SAM 2 also provides object tracking over multiple frames, which helps prevent duplicate object detection.
- We used GPS, LiDAR and camera data to approximate the location of the object.
- Qwen2.5 VL: the masked images are then prompted to Qwen, which creates a report for the object.
- PDF is generated using the FPDF library for Python.
The source code is available here.
Best Regards,
Ekam
Team Robocon IITR
