Visual Shelf Monitoring

Visual Shelf Monitoring

Image recognition algorithms analyze images streaming from retail store cameras, detecting (mis)placement of products, verifying these against target planograms and promotion compliance diagrams to deliver a real-time action stream for restocking, rearrangement and optimization of shelf product assortment. International media coverage and 4 patents filed.

In-store cameras and visual object recognition algorithms are used to continuously monitor in an automated way the presence, misplacement, and in particular, the lack of products on supermarket shelves. This allows retailers to avoid out-of-stock products, which otherwise incur revenue losses and cause customer dissatisfaction. In addition, consumer goods companies can monitor that their products are displayed according to negotiated agreements. The use of cameras allows for tag-less monitoring of products and requires no modification to the shelf infrastructure other than the deployment of a camera network. Research focused on practical deployment issues and algorithmic improvements to boost performance.

For retailers (and ultimately the manufacturers whose goods they sell), keeping supermarket shelves stocked is a vital part of running a successful operation. Empty or disarranged shelves mean lost sales as customers either cannot find a particular product or are not prompted to make an impulse buy. Retailers also have contractual obligations with manufacturers to comply with an agreed planogram (arrangement of products on a shelf). Monitoring shelves has always been an expensive— and inefficient—manual process for retailers, requiring stock clerks to monitor shelves throughout the day. Manual monitoring is prone to error, and stock outages can go undetected for long periods of time. Maintaining safety stocks is no solution because it ties up capital in stock and, ironically, can make out-of-stock monitoring seem less urgent. Similarly, using historical data to predict demand is unreliable as it does not account for the variable, contingent factors that affects demand such as holidays, heat waves and so on.
Furthermore, solutions based on radio frequency identification that automatically match incoming and outgoing stock are not yet in general use, especially at shelf level. In any event, radio frequency identification will require shelf modification to accommodate readers as well as the tagging of individual products. It may therefore prove to be uneconomic for retailers when it comes to in-store monitoring.

We crafted an innovative solution that will help retailers solve the problem of shelf monitoring.

How the Visual Shelf Monitoring Solution works

The Visual Shelf Monitoring Solution uses video cameras and state- of-the-art object recognition technology. Using our software, the video feeds are continuously monitored against the original planogram. Any anomalies—such as out-of-stock or misplaced items—trigger an alert to the computers (and even cell phones) of designated staff members. These alerts can then be acted upon to rectify the situation virtually as it happens. It allows retailers to automate the chore of shelf monitoring while increasing accuracy.

Many, many assets

A full range assets have have been delivered for this project covering all aspects of product development: software, hardware, media documents, patented ideas, videos, full-scale demos, and small transportable demos.

Software assets

Software was developed that integrated the object recognition algorithms on servers, stand-alone laptops, mobile phones, and web sites. The main application pulls in the live video feed from any camera, analysis the images, detects missing and misplaced products, and uses augmented reality to display where and what is missing. The solution also includes tools for crawling web sites for images, managing the image databases, making planograms, and help position cameras for maximal shelf coverage.

Workshop assets

Room-sized demos of the Visual Shelf Monitoring Solution is demoed in Europe, the States, and in Asia (see the first image above), including an installment at the Fontys University in the Netherlands where it became part of their teaching curriculum. Transportable suitcase versions were also developed for conferences, remote client visits, and smaller demo locations. Alerts can be shown remotely on iPhones or any web capable device.

Media coverage

We lost count of the number of documents in which this project has been discussed : white papers, thought leadership documents, presentations, flyers, websites.


  • Determination of inventory conditions based on image processing Filed 2007. Serial no. 11/849,177
  • Detection of stock out conditions based on image processing Filed 2007. Serial no.11/849,180
  • Planogram extraction based on image processing Patent #20090059270, 2007. Serial no. 11/849,165
  • Determination of product display parameters based on image processing Patent #20090063306, 2007 Serial no. 11/849,171

To find out more, feel free to leave or comment below or to contact me directly.

Leave a Comment