8. Camera use-Opencv

The Lubancat RK series boards support the use of common sensors such as cameras. The boards all have 24pin mipi csi interfaces. The camera used in the test is ov5648. This chapter will use the opencv-python library to use the camera and perform simple processing.

提示

If there is no special mention, the tutorials in this book are based on Python 3.8.10 version (the mirror system is Ubuntu20.04) for experiments and explanations. To learn the content of this chapter, you need a camera hardware device, please prepare by yourself.

8.1. Introduction to opencv-python library

OpenCV is a classic dedicated library in computer vision. It supports multi-language, cross-platform, and powerful. It was founded by Gary Bradsky in Intel in 1999 and released its first version in 2000. OpenCV now supports numerous algorithms related to computer vision and machine learning and is expanding day by day. OpenCV supports various programming languages, such as C++, Python, Java, etc., and is available on different platforms, including Windows, Linux, OS X (the system has been renamed macOS), Android, and iOS. High-speed GPU operation interfaces based on CUDA and OpenCL are also under active development.

OpenCV-Python is a Python-based library, or OpenCV’s Python application interface, designed to solve computer vision problems, combining the best features of the OpenCV C++ API and the Python language.

8.2. Add camera resource to Lubancat board

In this section of the experiment, the LubanCat 2 board is used as an example for demonstration. Before powering on, connect the camera to the mipi csi interface. After the system starts:

# Enter the following command in the terminal to view the camera device resources:
ls /dev/video*

Check the camera device resource example on the board, for reference only:

More camera usage references under linux system “Camera” . This chapter will introduce tools such as v4l-utils, and you can view the relevant information of the camera device.

8.3. Related tools and library installation

We will use some tools in the experiment, enter the following command to install (if installed, it is not necessary):

1、Install some dependent libraries or tools:

# Enter the following command in the terminal:
sudo apt update
sudo apt -y install python3-pil python3-numpy python3-pip git wget

# Libraries that may be required, some may be installed by default
sudo apt install libavcodec-dev libavformat-dev libswscale-dev
sudo apt install libgstreamer-plugins-base1.0-dev libgstreamer1.0-dev

Just wait for the installation to complete.

2、Install the opencv-python library:

Install directly using the command (recommended):

# The python3-opencv package can be installed using the apt command:
sudo apt-get install python3-opencv

# Or use pip to specify the opencv version to install:
pip3 install opencv-python==4.7.0.68

To get the whl file installation, you can go here to download the whl file corresponding to the python version and architecture

# For example, the following whl file can be installed using the command:
pip3 install opencv_python-4.5.4.60-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl

Or download the source code to compile and install, you can refer to this link .

提示

If you use the Debian10 system, you need to use opencv_python to control the camera, and you need to install a higher version of opencv_python and pip3. Use the command (update pip3 and install whl with root authority): sudo pip3 install --upgrade pip, and then choose the above method to install opencv_python.

3、Verify that the installation was successful:

Open an interactive Python terminal (or IPython):

cat@lubancat:~$ python3
Python 3.8.10 (default, Nov 14 2022, 12:59:47)
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2 as cv
>>> print(cv.__version__)
4.2.0
>>>

8.4. Camera to take pictures

Companion code ‘io/camera/camera_photo.py’ file content

import os
import cv2

# frame width and height
width = 640
height = 480

num = 1

# Create a VideoCapture object and open the system's default camera (you can also open the video or the specified device)
cap = cv2.VideoCapture(0)

# Can't open camera
if not cap.isOpened():
    raise RuntimeError('Could not open camera.')

# Set frame width and height
cap.set(cv2.CAP_PROP_FRAME_WIDTH, width)
cap.set(cv2.CAP_PROP_FRAME_HEIGHT, height)

while(cap.isOpened()):

    # Returns two parameters, ret indicates whether it is opened normally, frame is an image array
    ret,frame  = cap.read()

    # The window displays, named Lubancat_Camera_test
    cv2.imshow("Lubancat_Camera_test", frame)

    # Delay 1ms, and return the value val according to the keyboard input, which is the keyboard connected to the board
    val = cv2.waitKey(1) & 0xFF

    # The keyboard input 's' is captured, and the picture is saved to the current directory
    if val == ord('s'):
        print("=======================================")
        # The first parameter is the name of the picture to be saved as, and the second parameter is the image to be saved, in jpeg format
        cv2.imwrite("photo" + str(num) + ".jpg", frame)
        print("width = ", width)
        print("height = ", height)
        print("success to save photo: "'photo' + str(num)+".jpg")
        print("=======================================")
        num += 1

    # If the key 'q' is detected, exit, it is the keyboard connected to the board
    if val == ord('q'):
        break

# Release camera
cap.release()
# Close the window
cv2.destroyAllWindows()

Use the mirroring system with desktop, connect the keyboard and screen, run the program on the board system, and use the camera to read the above notes in detail. Run the program with the command:

sudo python3 camera_photo.py

At this time, a pop-up window will appear on the screen. When the keyboard presses “s”, the picture will be saved to the current directory, and the installation of “q” will exit the program. Run example:

8.5. Camera capture, record video

Each frame in the video represents an image, and we record the video by capturing a frame of image and compressing and saving it.

Companion code ‘io/camera/camera_video01.py’ file content

import os
import cv2

# frame width, height, frame rate
width = 640
height = 480
fps = 25.0

# Create a VideoCapture object and open the system default camera
video = cv2.VideoCapture(0)

# Set frame width and height
video.set(cv2.CAP_PROP_FRAME_WIDTH, width)
video.set(cv2.CAP_PROP_FRAME_HEIGHT, height)

# Define the writing format of the video file, the uncompressed YUV color coding type (the video file saved in this format is very large, you need to pay attention), some formats may not be supported, the encoder fails, and related libraries need to be installed
# fourcc = cv2.VideoWriter_fourcc(*'MJPG')
fourcc = cv2.VideoWriter_fourcc('I','4','2','0')

# It is used to save multiple images into video files. The first parameter is the name of the video file to be saved, and the second function is the codec code
# The third parameter is the frame rate of the saved video, and the fourth parameter is the size of the saved video file, which must be the same size as the image
# out = cv2.VideoWriter('output.mp4',fourcc, fps, (width,height))
out = cv2.VideoWriter('output.avi',fourcc, fps, (width,height))

while(video.isOpened()):
    ret, frame = video.read()
    if ret is True:
        # window displays, named Lubancat_Camera_video
        cv2.imshow('Lubancat_Camera_video',frame)

        # Store the captured image, the saved video has no sound
        out.write(frame)

        # Delay 1ms, if the keyboard presses "q", exit, it is the keyboard connected to the board
        if cv2.waitKey(1) & 0xFF == ord('q'):
            break
    else:
        print("Unable to read camera!")
        break

# Release resources
video.release()
out.release()
cv2.destroyAllWindows()

Run the program, it will capture and record, and the video will be saved in the current directory. See the note above for the detailed process. Exit when the keyboard presses “q”, or directly interrupt the running of the program.

To play video, you can also use cv2.VideoCapture, refer to the following:

Companion code ‘io/camera/camera_video02.py’

import cv2

# Create a VideoCapture object and select the video file to play
cap = cv2.VideoCapture('output.avi')

while(cap.isOpened()):
    ret, frame = cap.read()

    # Window displays, namedLubancat_Camera_video
    cv2.imshow('Lubancat_Camera_video',frame)

    # Delay 1ms, if the keyboard presses "q", exit, it is the keyboard connected to the board
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

# Release resources
cap.release()
cv2.destroyAllWindows()

8.6. Reference

https://docs.opencv.org/4.6.0/dc/d4d/tutorial_py_table_of_contents_gui.html