• Prasanna G
• Suhas K M
• Siri G
• Syed Furkhan Ahmed
• Thejas R

This paper presents a novel multimodal framework for real-time object detection and context-aware feedback, integrating cutting-edge deep learning techniques in computer vision and natural language processing. The system is specifically designed to enhance scene understanding for assistive technologies and autonomous platforms, with a particular focus on accessibility for visually impaired users. The proposed architecture leverages YOLOv8 for high-speed, real-time object detection and BLIP (Bootstrapping Language-Image Pre-training) for generating contextually rich visual captions. To address the limitations of generic captioning models in assistive contexts, BLIP is fine-tuned using Low-Rank Adaptation (LoRA) on the VizWiz dataset—comprising images taken by blind users and paired with human-annotated descriptions. This domain-specific adaptation significantly improves caption accuracy and relevance in complex, real-world environments. A key innovation of the system is a conditional activation mechanism that dynamically triggers the captioning module based on object motion, count variation, or manual user input. This strategy minimizes redundant processing, enhances computational efficiency, and supports deployment on edge devices with limited resources. The framework is implemented with a Streamlit-based interface supporting diverse input modalities, including static images, video streams, and live webcam feeds. Additional features include customizable detection thresholds, persistent object tracking with unique identifiers, real-time caption generation, and optional audio feedback via text-to-speech synthesis. The proposed system demonstrates a practical and scalable approach to bridging the gap between visual perception and semantic interpretation. Experimental results confirm its effectiveness in delivering timely, relevant, and context-sensitive feedback, underscoring its potential for integration into assistive technologies and real-time robotic perception systems.

Object Detection, Context Aware Feedback, YOLOv8, BLIP, Transfer Learning, LoRA (Low-Rank Adaptation), Visually Impaired, Assistive Technology, Multimodal System, Visual Language

"Real Time Object Detection And Context Aware Feedback Using Deep Learning", International Journal of Emerging Technologies and Innovative Research (www.jetir.org), ISSN:2349-5162, Vol.12, Issue 5, page no.i767-i789, May-2025, Available :http://www.jetir.org/papers/JETIR2505988.pdf

"Real Time Object Detection And Context Aware Feedback Using Deep Learning", International Journal of Emerging Technologies and Innovative Research (www.jetir.org | UGC and issn Approved), ISSN:2349-5162, Vol.12, Issue 5, page no. ppi767-i789, May-2025, Available at : http://www.jetir.org/papers/JETIR2505988.pdf