Understanding computer vision and its application in electronics

Ever since robotics could reliably automate many human tasks, engineers have been diligently trying to add the five senses to these capabilities — as in sight, sound, taste, smell, and touch. The most significant so far is vision.

However, how people see and how machines “see” is quite different. The human eye is a complex organ that functions in unity with the brain. Machines can only capture images and, more specifically in terms of sight, a series of timed images.

In this case, a machine’s camera is an important electronic sensor. Combined with the intelligence of a processor or controller, cameras can be used to design several useful applications.

Additionally, many camera sensors can be interfaced with Arduino, Raspberry Pi and other single-board computing platforms. This means you can use computer vision in your electronics and robotics projects.

What is computer vision?
Computer vision is an inter-disciplinary scientific field that covers the ability of computers to recognize and “understand” images and digital videos.

Since it’s currently next-to impossible to perfectly replicate the actual working of human eyes, computer vision is largely focussed on application-based problem solving using captured images or digital videos. To work effectively, it often incorporates artificial intelligence, machine learning, algorithms, and other specialized methods.

Although computer vision aims to reproduce the abilities of human eyes, it’s currently limited to extracting useful descriptions from images such as objects, texts, paths, or 3D models.

How is computer vision used?
Computer vision is used for detection or recognition of objects using images and digital videos. It can also be used to track objects or trace paths using visual data. The detection or recognition of objects is, typically, used to solve application-specific problems.

Much like in a gesture-recognition system, it can receive commands through the hand gestures of a user.

For example:

In a surveillance system, computer vision can detect the movement of a potential intruder or even recognize a suspect.
In a robotics application, it can find different pathways, recognize target objects, avoid obstacles, analyze the environment, and more.
In automation, it can be used to add vision-based artificial intelligence for quality control or efficiency.
As a simple application, such a vision system can maintain a record of images or digital videos so the computer recognizes and tracks its surroundings.

Nearly all of the applications computer vision is used for in robotics is to solve a problem by using visual data.

The system
A computer vision system uses electronics and requires a camera sensor to capture images and videos. It often also uses sensors, a processor or computing platform (for programmatic analysis of the images), a controller, and actuators.

The controller and actuators are key to accomplishing the programmed tasks, based on the analysis of images and videos.

Function
Designing a computer vision system involves an embedded design that uses a camera sensor and image processing for decision making. It’s fairly similar to designing any other embedded system

You may decide to add a camera to a robotics or electronics design for one of the following tasks:

1. Object segmentation: to detect pixels that make up an object in a captured image or video.
2. Object detection: to detect the object(s) in an image or video.
3. Object identification: to identify the object(s) in a captured image or video.
4. Object verification: to confirm the presence of an object using captured images or video.
5. Object classification: to classify the object(s) in a captured image — a part of recognizing a particular object or class of objects.
6. Object landmark detection: to track the position of an object in captured images or video.
7. Object recognition: to recognize an object using image processing techniques, machine learning, and/or artificial intelligence.

The use of computer vision in any electronics or robotics application/project can be broken down into these steps… embedded system

You may decide to add a camera to a robotics or electronics design for one of the following tasks:

1. Capturing images and digital videos. This involves interfacing a camera sensor or module to a computing platform. The camera modules come with different capabilities, such as 2D/3D, RGB/greyscale/binary, resolution, frame rate, etc.

Camera modules offer different options for interfacing, such as SPI, MIPI, and USB. The camera is only used to capture images or a timed series of images. It can be analog or digital and, typically, a digital one is used for computer vision applications.

2. Pre-processing images. Once a raw image or digital video is captured with the camera, the computing system must perform filtering and pre-processing (some is initially done by the camera). This is because the captured images might be blurry due to vibrations, movement, or other environmental factors.

It is extremely important to extract the required visual information from the captured images via proper pre-processing for accurate visual data.

Pre-processed images offer a lot of data. While a regular sensor, such as for temperature or humidity, provides data that’s a few bytes long, a single frame or color image (say, of 640×480 resolution) entails 640*480*3 = 921600 bytes of data. Therefore, the computing system must be capable of handling a large amount of data.

3. Image processing. Image processing is key in the development of computer vision projects. The purpose of adding computer vision is to extract meaning from the captured images. This requires various image-processing techniques and algorithms to extract the features from a single image, a set of images, or a series of timed images.

Often, inter-frame information must be extracted, such as for object landmark detection. And, one task that’s essential in any computer vision application is object segmentation. It’s used for object detection, verification, recognition, landmark detection, classification, and/or object recognition.

4. Post-processing. After image-processing techniques extract meaningful information from the image(s), the data may be further processed for higher-level feature extraction or pattern recognition.

5. Detection/recognition. Lastly, the computing system must be able to detect or recognize the object(s). Recognition is a complex process that uses classification, pattern recognition, and identification.

6. Decision making. The purpose of the visual data collection is, ultimately, to perform a task. After the system detects and recognizes the object(s), the machine decides the next step. For instance, a surveillance system may perform facial recognition using computer vision and, then, unlock a door for an authorized visitor.

7. Artificial Intelligence and machine learning. Computer vision systems use multi-disciplinary fields that might include image processing, artificial intelligence, machine learning, or deep learning.

Camera sensors
Typical camera modules used with Arduino or other microcontroller platforms include: OV7670, OV9655, Arducam MT9D111, Arducam MT9M001, Arducam OV5642, OV2640, Yosoo camera module, and Pixy smart vision sensor.

For Raspberry Pi, these cameras are often used: OV9281, OV2311, IMX 135, OV7251, IMX298, AR1820HS and official raspberry pi camera can be used.

For Jetson Nano and Xavier NX, consider: the IMX477, IMX219, OV9281, OV7251, and OV2311.

Working with a computer vision application…

Using OpenCV
The Open Source Computer Vision Library (OpenCV) is a popular library that helps users to apply various image processing algorithms and techniques. It’s a game-changer that’s simplified sophisticated image processing tasks.

The library can be used with several programming languages including C++, Java, and Python. OpenCV can be run on a desktop, Raspberry Pi, or any single-board computer.

Using Arduino
On its own, Arduino is incapable of performing image processing and implement computer vision applications. However, it can be combined with single-board computers and desktop systems to complement a computer vision project.

Arduino can also be used to interface additional sensors, such as temperature and humidity, or to control actuators like motors, servos, or relays.

Using Raspberry Pi
Raspberry Pi is capable of running a full-fledged computer vision application. It’s possible to install and run OpenCV in Raspberry Pi and design Python, Java, or C++ programs to implement a computer vision project.

Using a Jetson platform
Along with the image processing, the Jetson platform can add artificial intelligence and machine learning to a computer vision project. Nano and Xavier NX are its most popular boards.

Applications
Some of the more popular computer vision applications include: