Gesture recognition with the help of Convolutional Neural Network (CNN)

1. The images to be classified must Square cut to fit the hand in the center be.
2. The Background had to be monochrome and reflect a correspondingly high contrast to the skin color (e.g. a white wall in the background).
3. It always had to be a right hand which projects straight into the image from below and represents the number of fingers in the way presented in the training data set.

If these rules were adhered to, we were able to achieve an accuracy of around over 90 percent can be achieved. This is a very good result, considering that a simple artificial neural network with two relatively small hidden layers was used here.

In this article, I would like to show the approaches we have used to successfully implement our model:

1. to improve, i.e. to achieve greater accuracy and
2. to generalize, i.e. to be able to classify both hands in different positions, with a wider image section and any background.

Exchange of the architecture used: Convolutional Neural Network (CNN)

In the last decade Convolutional Neural Networks (CNN) - neural networks - are making their breakthrough in many areas. Classification tasks with image and audio data and were able to qualify for these as State-of-the-art method establish.

A CNN is made up of several components:

1. Filters that are referred to as convolutional layers
2. aggregation layers, which are referred to as pooling layers, and
3. classic neural networks, which are referred to as dense or fully connected layers.

Convolutional (1.) and Pooling (2.) layers are repeated alternately and are then completed by one or more Dense/Fully Connected (3.) layers at the end.

The convolutional layers represent the Pre-processing of input data and filter certain properties of the image such as edges, certain areas, colors or structures. For this purpose, a filter (also known as a sliding window) of a fixed size (e.g. 3 x 3 pixels) scans over the input image. This filter attempts to reduce the input to certain properties. By using several filters, different properties can be filtered out.

However, as each filter generates its own modified image, the amount of data to be processed increases significantly with each additional convolutional layer in the convolutional neural network. For this reason, pooling layers are used to condense the data. For this purpose, several pixels are reduced to one, e.g. by reducing a 4 x 4 pixel matrix to one pixel by extracting the maximum value from this matrix.

 

This two-stage procedure is based on the behavior of the visual cortex of the human brain, which is responsible for visual perception.

This can be vividly visualized, for example, by training with faces:

 

Another very vivid example can be found here.

Implementation of the Convolutional Neural Network with the Keras framework

To implement the CNN Keras used. Keras is a high-level API for Tensorflow. It allows Quick experimentation with different network configurations and provides some Features for abstracted data pre-processing.

Finally, we decided on a convolutional neural network with four convolutional layers with 32, 64, 128 and 128 filters respectively, followed by a neural network with 512 neurons. With this configuration, we were able to increase the accuracy to over 95 percent raise.

Enlarging the training data set for the Convolutional Neural Network

Finally, we turned our attention to the Generalization of the model. To do this, we need one thing above all: data, data, data.
So the first step was to collect more data. This was done via a Further data set (senz3d) and by Pictures that we have taken ourselves. The images added in this way no longer contained a monochrome, calm background and showed both hands. They were also not necessarily large or centered but shown in different positions. They were also no longer just square.

With this approach, we increased the data set from around 1000 images to around 4000 images. However, as this number is still below the typical minimum for training deep neural networks is the reason why we also used the Image Augmentationwhich Keras also supplies. In this context, this means the Artificial duplication of the data with slight modifications, or the artificial generation of new data.

For example, you can create a left hand from a right hand by mirroring it or by rotating, zooming in, cropping and distorting it Simulate different shooting angles.
All this leads to many new images being generated, each with slight variations. However, each of these artificially generated images is still relevant to the Training of the Convolutional Neural Network suitable, as it depicts a new situation each time (left hand, right hand, angle of the hand,...) and thus helps the Improve the accuracy of the trained model.

On another self-collected test data set with moderate difficulty (relatively calm, but not monochrome background, hand centered from below into the image, ...), the accuracy of just under 30 percent with the old architecture could be reduced to over 85 percent increase.

Since this extension meant that the data was no longer "as clean" as the original data set and the network itself has to find the hands in the images, recognize them as such and classify them using this method, the Very satisfactory result.

The trained model now replaces the previous demo version and can be tested here:

Click here for the demo application

Summary

To improve gesture recognition in our demo version, we used a special type of artificial neural network, so-called Convolutional Neural Networks who have their special Strength in the processing of image and audio data have. At the same time, we have the Number of training data records greatly increased. Both have led to the fact that we have a large Improvement in quality of gesture recognition. The application now also makes it possible to correctly classify hand signals in more complex situations.