Classificação e rastreamento de itens em uma esteira móvel utilizando redes convolucionais e processamento de imagens

Abstract

Given the prosperity of the modern world, there is a growing need to reduce the time spent on trivial chores. In the context of buying groceries, recent studies point out that one of the most relevant factors on the buyer’s experience, that reflects on sales and revenue, is the time spent in queues. The work in question aims to describe the creation of a computer vision and deep learning prototype, to be installed next to a camera suspended on a mobile supermarket conveyor belt. It will be responsible for detecting, classifying, tracking and counting of all passing items. The video stream is processed in real time, and upon detecting the passage of a specific item, the final purchase bill is increased. As there would be no human interference, the process tends to simplify, make cheaper and speed up supermarket checkouts. Among the technologies explored is “state of the art” convolutional neural networks (CNN), especially YOLO v4 tiny and YOLO v5 small, as well as some more consolidated ones such as OpenCV for image processing or Roboflow for database augmentation. At the end of the experiment, it was possible to develop a model that had up to 77% of average precision (mAP@[0.5:0.95]) for two items on a treadmill, using a model trained in a hybrid dataset, composed of images collected in vitro and images generated through a simulator, in addition to a graphical interface responsible for viewing the processed video feed, which also allows manipulation of hyperparameters from the CNN, tracker and item counter.