1. Video encoding and decoding - based on mpp library

1.1. Introduction to MPP

The Media Processing Platform (MPP) provided by Rockchip is a universal media processing software platform suitable for Rockchip microchip series. The platform shields application software from the complex underlying processing related to the chip. Its purpose is to shield the differences between different chips and provide users with a unified video media processing interface (Media Process Interface, abbreviated MPI). Features provided by MPP include:

video decoding	H.265 / H.264 / H.263 / VP9 / VP8 / MPEG-4 / MPEG-2 / MPEG-1 / VC1 / MJPEG / AV1
video encoding	H.265 / H.264 / VP8 / MJPEG
video processi	Video copy, scaling, color space conversion, field video deinterlace

The following is the GitHub link to RK’s official MPP documentation, which contains the Chinese/English MPP development guide.

https://github.com/rockchip-linux/mpp/tree/develop/doc

1.2. Obtain and compile the RKMPP library

1.2.1. Test environment

Test board: LubanCat 1 (LubanCat 4 is similar)

Test operating system:Ubuntu20.04

Kernel version:Linux 4.19.232

RK official MPP library address: https://github.com/rockchip-linux/mpp

1.2.2. Install related dependency tools

sudo apt update
sudo apt install -y git cmake

1.2.3. Pull RK official MPP repository

git clone https://github.com/rockchip-linux/mpp.git

1.2.4. Compile

Enter the corresponding compilation path of aarch64.

cd mpp/build/linux/aarch64/

Modify the cross-compilation configuration file and specify the compilers gcc and g++ (usually the default is fine).

vim arm.linux.cross.cmake

Run the bash script and compile (the compilation process takes about 15 minutes).

./make-Makefiles.bash
make

After compilation, you will find that there are many more files in the directory.

Enter the test directory, which contains some compiled and generated test programs.

1.3. Video decoding

The decoder demo is the mpi_dec_test series of programs, including single-threaded mpi_dec_test using the decode_put_packet and decode_get_frame interfaces, multi-threaded mpi_dec_mt_test and multi-instance mpi_dec_multi_test.

1.3.1. Test environment

Test board: Luban Cat 1

Test operating system: Ubuntu20.04

Kernel version: Linux 4.19.232

1.3.2. Command parameters of mpi_dec_test

1.3.2.1. Check the command parameters of mpi_dec_test in the terminal

Open two terminals and enter the following command in one terminal to monitor the log output.

sudo tail -f /var/log/syslog

Another terminal executes the mpi_dec_test program.

mpi_dec_test

After executing the test program, the following help document will be printed in the log:

The help document can be divided into two parts: one is the command parameter description of mpi_dec_test; the other is the protocol type description of the code stream file.

1.3.2.2. Command parameter description of mpi_dec_test

The description of the command parameters is as follows:

-i	Input stream file.
-o	The output image file.
-w	Image width in pixels.
-h	Image height in pixels.
-t	The protocol type of the code stream file.
-f	Image color space format and memory arrangement, the default is NV12.
-n	Maximum number of decoded frames. If the code stream is long during testing, only the first n frames can be output.
-s	The number of MPP instances, the default is 1.
-v	Log options: q is the silent flag; f is the fps display flag.
-slt	Output the verification file corresponding to the frame.
-help	Open the help document.

小技巧

1. Among the command parameters of mpi_dec_test, the input file (i) and code stream type (t) are mandatory parameters. Other parameters such as output file (o), image width (w), image height (h) and decoded frame number (n), etc. are optional parameters and can be configured according to different testing requirements.

2. In the command parameters of mpi_dec_test, the check file (slt) corresponding to the output frame converts the output frame data into the corresponding cyclic redundancy check code (see utils/utils.c for specific logic). The size of the verification file is often only a few kB. In the SLT test of the chip, converting the comparison of the output frame file into the comparison of the verification file can significantly shorten the test cycle.

1.3.2.3. Description of code stream file protocol types supported by MPP decoding

Decoding types supported by MPP:

小技巧

The encoding formats (t) of input files supported by the MPP library are MPEG2/4, H.263/4/5, VP8/9 and JPEG, etc. The numbers after the id are parameter values corresponding to different encoding formats. Parameter values come from the definition of OMX. It is worth noting that the parameter values of HEVC and AVS formats are significantly different from those of other formats.

1.3.3. DecodingDemo

Here we take decoding the video “01-什么是鲁班猫.mp4” as an example to demonstrate the decoding process.

小技巧

The mp4 decoding process is mainly divided into two steps. The first step is to convert mp4 into a pure video type that the mpp library supports decoding (such as: h264). The second step is to use the mpp library to decode the converted video.

1.3.3.1. Convert mp4 to h264

Convert mp4 to h264, here we use the FFmpeg tool. For other uses of the FFmpeg tool, please refer to the corresponding chapters.

sudo apt update && sudo apt install -y ffmpeg #安装ffmpeg工具
ffmpeg -i 01-什么是鲁班猫.mp4 -c:v libx264 01.h264

Among them, 01-什么是鲁班猫.mp4 is the source file name to be converted, and 01.h264 is the output file name.

1.3.3.2. h264 decoding

This step is mainly to decode the 01.h264 file. Open two terminals and enter the following command in one terminal to monitor the log output:

sudo tail -f /var/log/syslog

Another terminal executes the decoding program:

mpi_dec_test -i 01.h264 -t 7 -n 60 -o 01.yuv

提示

The function of the above command is to decode 01.h264 and save it as 01.yuv. Among them, -i indicates the input file, -t 7 indicates that the protocol type of the input stream file is H.264, -n 60 indicates decoding 60 frames, and -o indicates the output file.

Part of the decoding log output is as follows:

1.4. Video encoding

The encoder demo is the mpi_enc_test series of programs, including single-threaded mpi_enc_test and multi-instance mpi_enc_multi_test.

1.4.1. Test environment

Test board: LubanCat 1

Test operating system: Ubuntu20.04

Kernel version: Linux 4.19.232

1.4.2. Command parameters of mpi_enc_test

1.4.2.1. Check the command parameters of mpi_enc_test in the terminal

Open two terminals and enter the following command in one terminal to monitor the log output.

sudo tail -f /var/log/syslog

Another terminal executes the mpi_dec_test test program

mpi_enc_test

After executing the test program, the following help document will be printed in the log:

The help document can be divided into three parts: the first is the command parameter description of mpi_enc_test; the second is the protocol type description of the code stream file; the third is the image color space format and memory arrangement description.

1.4.2.2. Command parameter description of mpi_enc_test

The description of the command parameters is as follows:

Command parameters	Description.
-i	Input image file.
-o	Output stream file.
-w	Image width in pixels.
-h	Image height in pixels.
-hstride	The distance between two adjacent rows in the vertical direction, in bytes.
-vstride	The number of intervals between image components in line numbers, in units of 1.
-f	Image color space format and memory arrangement, the default is NV12.
-t	The protocol type of the code stream file.
-tsrc	Source stream format, only used when testing overall encoding and decoding performance.
-n	Maximum number of decoded frames. If the code stream is long during testing, only the first n frames can be output.
-g	Gop reference mode, corresponding to different TSVC code streams.
-rc	Rate control mode. 0:VBR; 1:CBR; 2:FIXQP; 3:AVBR.
-bps	Code rate constraint parameters. Command format: bps_target:bps_min:bps_max.
-fps	Input/output frame rate control, default is 30. This command parameter only describes the proportional relationship between the input frame rate and the output frame rate, and has nothing to do with the actual frame rate.
-qc	Quality Control.
-s	The number of MPP instances, the default is 1.
-v	Log options: q is the silent flag; f is the fps display flag.
-ini	Additional encoding configuration file ini (not yet effective).
-slt	Output the verification file corresponding to the code stream.

小技巧

1. Among the command parameters of mpi_enc_test, image width (w), image height (h) and code stream type (t) are mandatory parameters. Other parameters such as input file (i), output file (o), number of encoding frames (n), color space format and memory arrangement (f) are optional parameters. If no input file is specified, mpi_enc_test will generate a default colorbar image for encoding.

2. The command parameters of mpi_enc_test provide diverse code rate control solutions. Users can control the code rate of the output code stream through the code rate control mode (rc) and code rate constraint parameters (bps). The code rate control mode (rc) is divided into variable code rate mode (VBR), fixed code rate mode (CBR), qp modified code rate mode (FIXQP) and adaptive code rate mode (AVBR). The default mode is VBR; The code rate constraint parameter (bps) provides a reference for MPP internal configuration of code rate boundaries.

3. In the command parameters of mpi_enc_test, when the log option (v) is q, the MPP daily log is turned off; when the log option (v) is f, the average frame rate and current frame rate will be printed once every second.

In the command parameters of mpi_enc_test, the format of input/output frame rate control (fps) is:

-fps fps_in_num:fps_in_den:fps_in_flex/fps_out_num:fps_out_den:fps_out_flex

小技巧

Among them, in/out represents input/output respectively; num represents the numerator; den represents the denominator; If the flex parameter is 0, it means the frame rate is fixed, and if it is 1, it means the frame rate is variable. The default num and den for input and output are 30 and 1 respectively, that is, the default input/output frame rate is 30. This command parameter only describes the proportional relationship between the input frame rate and the output frame rate, and has nothing to do with the actual frame rate.

Among the command parameters of mpi_enc_test, quality control (qc) only takes effect when the output stream format is H.264, H.265, VP8 and JPEG. The command format is:

-qc qp_init/min/max/min_i/max_i

1.4.2.3. Description of code stream file protocol types supported by MPP encoding

Coding types supported by MPP:

小技巧

1. The encoding format (t) of input files supported by the MPP library is H.265/H.264/VP8/MJPEG, etc. The numbers after the id are the parameter values corresponding to different encoding formats.

1.4.2.4. Image color space format and memory arrangement description

The color space format of the image is divided into two categories: YUV and RGB. MPP supports multiple memory arrangement methods (f), and the number after the id is the parameter value corresponding to different memory arrangement methods. It is worth noting that the parameter values of YUV and RGB formats are significantly different.

1.4.3. Coding Demo

Here we take the 01.yuv file decoded above as an example to demonstrate the encoding process.

小技巧

Here we mainly demonstrate how to encode yuv files into h265 through the mpp library, and how to use ffmepg to convert h265 into mp4.

1.4.3.1. yuv encoded into h265

Open two terminals and enter the following command in one terminal to monitor the log output:

sudo tail -f /var/log/syslog

Another terminal executes the encoding program.

mpi_enc_test -i 01.yuv -w 1920 -h 1080 -t 16777220 -o 01.h265 -n 20

提示

The function of the above command is to encode 01.yuv and save it as 01.h265. Among them, -i means the input file, -w 1920 means the specified pixel width is 1920, -h 1080 means the specified pixel height is 1080, -t 16777220 means the protocol type of the output stream file is H.265, -n 60 means decoding 60 frames , -o means output file.

Part of the encoding log output is as follows:

1.4.3.2. h265 to mp4

Convert h265 to mp4, here we use the FFmpeg tool.

sudo apt update && sudo apt install -y ffmpeg #安装ffmpeg工具
ffmpeg -i 01.h265 -c:v libx265 -c:a aac -f mp4 0101.mp4

提示

Among them, 01.h265 is the H.265 video file name to be transcoded, 0101.mp4 is the MP4 file name after transcoding, -c:v libx265 encodes the video into H.265 format, -c:a aac encodes the audio into AAC format, -f mp4 specifies the output format as MP4.

1.5. Utilities

MPP provides some tool programs for unit testing, which can test the software and hardware platforms as well as the MPP library itself.

mpp_info_test	Used to read and print the version information of the MPP library. When reporting problems, you can attach the printed information.
mpp_buffer_test	Used to test whether the kernel’s memory allocator is normal.
mpp_mem_test	Used to test whether the memory allocator of the C library is normal.
mpp_runtime_test	Used to test whether some software and hardware runtime environments are normal.
mpp_platform_test	Used to read and test whether the chip platform information is normal.