Automated disease recognition plays a pivotal role in advancing smart AI-based agriculture and is crucial for achieving higher crop yields. Although substantial research has been conducted on Deep Learning (DL)-based automated plant disease recognition systems, these efforts have predominantly focused on leaf diseases while neglecting other parts like fruits. Diverse diseases affecting fruits significantly impact crop yield and the agro-economy as well as present the pressing need for a dedicated fruit disease recognition system. To address these issues, we propose an efficient DL architecture specially tailored for recognizing various fruit diseases with state-of-the-art performance. In addition, with a focus on Green AI, we designed the proposed method in an ultra-lightweight manner which not only reduces computational costs but also makes it deployable on memory-constrained edge devices, enhancing accessibility and practical applicability. Experimental evaluations on datasets of sugar apple, pomegranate, and guava diseases yielded exceptional test set accuracies of 99.15%, 99.03%, and 99.19%, respectively. The combination of high accuracy and lightweight architecture represents a significant advancement in developing low-cost, accessible AI-based smart agricultural systems, contributing to the broader field of smart agriculture.