5. Flower image classification–TensorFlow

TensorFlow is a symbolic mathematical system based on data stream programming (DataFlow Programming). It is widely used in programming implementation of various machine learning algorithms.

This chapter will briefly introduce a flower image classification task of TensorFlow, use TF.KERAS.SEQUENTIAL model to simply build a model, and finally convert it to the RKNN model to deploy the RK series board in Lubancat.

提示

Test environment: The Lubancat board uses Debian10, and the PC side is Ubuntu20.04. TensorFlow version 2.6.2, RKNN-Toolkit2 version 1.4.0.

5.1. Install TensorFlow environment

Use Ubuntu or Windows system to install TensorFlow in PC. The following example is in Ubuntu20.04 system:

# Install Python and update PIP, you need to use Python 3.6-3.9 and PIP 19.0 and higher versions
sudo apt update
sudo apt install python3-dev python3-pip python3-venv
sudo python3 -m pip install pip --upgrade

# Generally, a virtual environment is created
sudo python3 -m venv  .tensorflow_venv

# Order the latest stable version Tensorfolw directly, or you can specify the next version (GPU and CPU merger)
pip3 install tensorflow

For detailed installation steps and requirements, please refer to Tensorflow tutorial 。

提示

In the Windows system (Windows64 -bit operating system), you need to install Python, VC ++, Anaconda <https://www.anaconda.com/products/distribution> _, etc. If the NVIDIA GPU is required, the GPU driver is required, CUDA, CUDNN. The version between them corresponds to reference. Here is <https://tensorflow.google.cn/install/source_windows?hl=en#gpu> _. Between the graphics card driver and the CUDA version, please refer to the following <https://docs.nvidia.com/cuda/cuda-toolkit-rease-notes/index.html>. There are many installation tutorials on the Internet. According to your own equipment, search for a new date document.

5.2. Image classification

The following will introduce a simple example to classify flower images using TF.KERAS.SEQUENTIAL model. For details, please refer to https://tensorflow.google.cn/tutorials/images/classification#overfitting

5.2.1. Prepare data set

Download data set, including 3700 images, use tf.keras.utils.image_dataset_from_directory to divide the data 80% for test sets, and the remaining 20% is used to verify the test set.

tensorflow_classification.py(Reference supporting example)
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
# Retrieve data
import pathlib
dataset_url = "https://storage.googleapis.com/download.tensorflow.org/example_images/flower_photos.tgz"
data_dir = tf.keras.utils.get_file('flower_photos', origin=dataset_url, untar=True)
data_dir = pathlib.Path(data_dir)

# Setting parameters
batch_size = 32
img_height = 180
img_width = 180

# Use tf.keras.utils to introduce the dividing data set, divided into training sets and verification sets
train_ds = tf.keras.utils.image_dataset_from_directory(
data_dir,
validation_split=0.2,
subset="training",
seed=123,
image_size=(img_height, img_width),
batch_size=batch_size)

val_ds = tf.keras.utils.image_dataset_from_directory(
data_dir,
validation_split=0.2,
subset="validation",
seed=123,
image_size=(img_height, img_width),
batch_size=batch_size)

class_names = train_ds.class_names
#print(class_names)

# Process data
normalization_layer = layers.Rescaling(1./255)
train_ds = train_ds.map(lambda x, y: (normalization_layer(x), y))
val_ds = val_ds.map(lambda x, y: (normalization_layer(x), y))
num_classes = len(class_names)

提示

Some API interface descriptions and use of TensorFlow are referenced <https://tensorflow.google.cn/api_docs/python/tf>

5.2.2. Construction and training model

Keras Sequential model construction: It mainly contains three volumes (TF.KERAS.Layers.conv2D), each of which has a largest pooling layer (TF.karas.Layers.maxpooling2D) in each convolutional block. Finally, there is a full connection layer (TF.KERAS.Layers.Density), all use RELU as the activation function.

tensorflow_classification.py(Reference supporting example)
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
# Construct a network structure
model = Sequential([
    layers.Rescaling(1./255, input_shape=(img_height, img_width, 3)),
    layers.Conv2D(16, 3, padding='same', activation='relu'),
    layers.MaxPooling2D(),
    layers.Conv2D(32, 3, padding='same', activation='relu'),
    layers.MaxPooling2D(),
    layers.Conv2D(64, 3, padding='same', activation='relu'),
    layers.MaxPooling2D(),
    layers.Flatten(),
    layers.Dense(128, activation='relu'),
    layers.Dense(num_classes)
])

# Configure the optimizer ‘adam’, loss function, evaluation indicator
model.compile(optimizer='adam',
            loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
            metrics=['accuracy'])

# Check the network structure of each layer
model.summary()

# Training model
epochs=10
model.fit(
    train_ds,
    validation_data=val_ds,
    epochs=epochs,
)
Check the network structure of each layer
Model: "sequential_1"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
rescaling (Rescaling)        (None, 180, 180, 3)       0
_________________________________________________________________
conv2d (Conv2D)              (None, 180, 180, 16)      448
_________________________________________________________________
max_pooling2d (MaxPooling2D) (None, 90, 90, 16)        0
_________________________________________________________________
conv2d_1 (Conv2D)            (None, 90, 90, 32)        4640
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 45, 45, 32)        0
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 45, 45, 64)        18496
_________________________________________________________________
max_pooling2d_2 (MaxPooling2 (None, 22, 22, 64)        0
_________________________________________________________________
flatten (Flatten)            (None, 30976)             0
_________________________________________________________________
dense (Dense)                (None, 128)               3965056
_________________________________________________________________
dense_1 (Dense)              (None, 5)                 645
=================================================================
Total params: 3,989,285
Trainable params: 3,989,285
Non-trainable params: 0
_________________________________________________________________

92/92 [==============================] - 65s 644ms/step - loss: 1.4799 - accuracy: 0.3689 - val_loss: 1.1546 - val_accuracy: 0.4877
Epoch 2/10
92/92 [==============================] - 52s 560ms/step - loss: 1.0406 - accuracy: 0.5777 - val_loss: 1.0439 - val_accuracy: 0.5913
Epoch 3/10
92/92 [==============================] - 51s 555ms/step - loss: 0.8372 - accuracy: 0.6764 - val_loss: 1.0274 - val_accuracy: 0.6022
Epoch 4/10
92/92 [==============================] - 51s 552ms/step - loss: 0.6344 - accuracy: 0.7698 - val_loss: 1.0563 - val_accuracy: 0.5831
Epoch 5/10
92/92 [==============================] - 51s 553ms/step - loss: 0.4266 - accuracy: 0.8539 - val_loss: 1.2350 - val_accuracy: 0.6104
Epoch 6/10
92/92 [==============================] - 51s 553ms/step - loss: 0.2695 - accuracy: 0.9087 - val_loss: 1.4331 - val_accuracy: 0.6213
Epoch 7/10
92/92 [==============================] - 51s 554ms/step - loss: 0.1561 - accuracy: 0.9533 - val_loss: 1.6564 - val_accuracy: 0.5981
Epoch 8/10
92/92 [==============================] - 51s 554ms/step - loss: 0.0725 - accuracy: 0.9785 - val_loss: 2.0034 - val_accuracy: 0.6281
Epoch 9/10
92/92 [==============================] - 51s 556ms/step - loss: 0.0440 - accuracy: 0.9888 - val_loss: 2.1280 - val_accuracy: 0.6076
Epoch 10/10
92/92 [==============================] - 51s 555ms/step - loss: 0.0194 - accuracy: 0.9973 - val_loss: 2.4841 - val_accuracy: 0.5886

Judging from the above training results, the training accuracy increased linearly over 99%over time, and the verification accuracy was about 60%. Too much difference between training accuracy and verification accuracy is overfitting phenomenon. To put it simply, the model can get better training data on training data than other data, but it does not achieve effect on the data set outside the training data, which means that the model is difficult to promote on the new dataset.

Generally, the reason for fitting is that the number of data set samples is too small, the data noise interference in the sample is too large, and the learning model is too complicated. Judging from the training here, it may be that the training dataset is too small, the model remembers noise, etc., and ignores the relationship between the real input and output, resulting in excessive fitting.

We extend the scale of the dataset here, carry out random flip, rotation, and zooming images at the Keras pre -processing layer, and use tf.keras.Layers.randomflip, tf.keras.Layers.randomrotation <https://tensorflow.google.cn/api_docs/python/tf/Layers/randomrotation> _ and and tf.keras.layers.randomzoom <https://tensorflow.google.cn/api_docs/python/tf/keras/randomzoom> _ interface. In addition, add the Dropout layer, and for neural network units, it is temporarily discarded from the network at a certain probability. Program examples are as follows:

tensorflow_classification.py(Reference supporting example)
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
# Build a network structure, add data_augmentation, and pre -processing layer
data_augmentation = keras.Sequential(
    [
        layers.RandomFlip("horizontal",
                        input_shape=(img_height,
                                    img_width,
                                    3)),
        layers.RandomRotation(0.1),
        layers.RandomZoom(0.1),
    ]
)

model = Sequential([
    data_augmentation,    # Add data_augmentation
    layers.Conv2D(16, 3, padding='same', activation='relu'),
    layers.MaxPooling2D(),
    layers.Conv2D(32, 3, padding='same', activation='relu'),
    layers.MaxPooling2D(),
    layers.Conv2D(64, 3, padding='same', activation='relu'),
    layers.MaxPooling2D(),
    layers.Dropout(0.2),   # Add Dropout layer
    layers.Flatten(),
    layers.Dense(128, activation='relu'),
    layers.Dense(num_classes, name="outputs")
])

Simple test results:

Epoch 1/15
2023-02-20 19:39:40.074247: I tensorflow/stream_executor/cuda/cuda_dnn.cc:369] Loaded cuDNN version 8202
92/92 [==============================] - 115s 759ms/step - loss: 1.3321 - accuracy: 0.4179 - val_loss: 1.0552 - val_accuracy: 0.5872
Epoch 2/15
92/92 [==============================] - 58s 635ms/step - loss: 1.0306 - accuracy: 0.5978 - val_loss: 0.9923 - val_accuracy: 0.6267
Epoch 3/15
92/92 [==============================] - 58s 635ms/step - loss: 0.9465 - accuracy: 0.6345 - val_loss: 0.9616 - val_accuracy: 0.6240
Epoch 4/15
92/92 [==============================] - 61s 661ms/step - loss: 0.8700 - accuracy: 0.6604 - val_loss: 0.8555 - val_accuracy: 0.6717
Epoch 5/15
92/92 [==============================] - 61s 665ms/step - loss: 0.7899 - accuracy: 0.6993 - val_loss: 0.8514 - val_accuracy: 0.6717
Epoch 6/15
92/92 [==============================] - 57s 620ms/step - loss: 0.7789 - accuracy: 0.6979 - val_loss: 0.7658 - val_accuracy: 0.6907
Epoch 7/15
92/92 [==============================] - 59s 640ms/step - loss: 0.7197 - accuracy: 0.7159 - val_loss: 0.7940 - val_accuracy: 0.6826
Epoch 8/15
92/92 [==============================] - 59s 644ms/step - loss: 0.6956 - accuracy: 0.7367 - val_loss: 0.7710 - val_accuracy: 0.6907
Epoch 9/15
92/92 [==============================] - 60s 657ms/step - loss: 0.6532 - accuracy: 0.7510 - val_loss: 0.7250 - val_accuracy: 0.7112
Epoch 10/15
92/92 [==============================] - 62s 672ms/step - loss: 0.6189 - accuracy: 0.7657 - val_loss: 0.7893 - val_accuracy: 0.7044
Epoch 11/15
92/92 [==============================] - 60s 652ms/step - loss: 0.6221 - accuracy: 0.7674 - val_loss: 0.6982 - val_accuracy: 0.7262
Epoch 12/15
92/92 [==============================] - 60s 651ms/step - loss: 0.5598 - accuracy: 0.7888 - val_loss: 0.6821 - val_accuracy: 0.7357
Epoch 13/15
92/92 [==============================] - 60s 649ms/step - loss: 0.5519 - accuracy: 0.7885 - val_loss: 0.7939 - val_accuracy: 0.7084
Epoch 14/15
92/92 [==============================] - 61s 667ms/step - loss: 0.5387 - accuracy: 0.7997 - val_loss: 0.8331 - val_accuracy: 0.6880
Epoch 15/15
92/92 [==============================] - 60s 653ms/step - loss: 0.5068 - accuracy: 0.8151 - val_loss: 0.6627 - val_accuracy: 0.7466

After Keras pre -processing layer and adding the Dropout layer. You can get the training model.

5.2.3. Test model

tensorflow_classification.py(Reference supporting example)
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
# Get the import picture
sunflower_url = "https://storage.googleapis.com/download.tensorflow.org/example_images/592px-Red_sunflower.jpg"
sunflower_path = tf.keras.utils.get_file('Red_sunflower', origin=sunflower_url)

img = tf.keras.utils.load_img(
    sunflower_path, target_size=(img_height, img_width)
)
img_array = tf.keras.utils.img_to_array(img)
img_array = tf.expand_dims(img_array, 0)

# Test
predictions = model.predict(img_array)
score = tf.nn.softmax(predictions[0])

# output the result
print(
    "This image most likely belongs to {} with a {:.2f} percent confidence."
    .format(class_names[np.argmax(score)], 100 * np.max(score))
)

5.2.4. TensorFlow Lite model

The trained Keras Sequential model, here uses tf.lite.tfliteconverter.from_keras_model to generate TensorFlow Lite model:

tensorflow_classification.py(Reference supporting example)
1
2
3
4
5
6
7
# Convert the model.
converter = tf.lite.TFLiteConverter.from_keras_model(model)
tflite_model = converter.convert()

# Save the model.
with open('model.tflite', 'wb') as f:
f.write(tflite_model)

TensorFlow Lite converter conversion workflow introduction reference tutorial

5.2.5. Model conversion and simulation test

Use Rknn-Toolkit2 to export the RKNN model:

rknn_transfer.py(Reference supporting example)
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
img_height = 180
img_width = 180
IMG_PATH = 'test.jpg'
class_names = ['daisy', 'dandelion', 'roses', 'sunflowers', 'tulips']

if __name__ == '__main__':

    # Create RKNN object
    #rknn = RKNN(verbose='Debug')
    rknn = RKNN()

    # Pre-process config
    print('--> Config model')
    rknn.config(mean_values=[0, 0, 0], std_values=[255, 255, 255], target_platform='rk3568')
    print('done')

    # Load model
    print('--> Loading model')
    ret = rknn.load_tflite(model='model.tflite')
    if ret != 0:
        print('Load model failed!')
        exit(ret)
    print('done')

    # Build model
    print('--> Building model')
    ret = rknn.build(do_quantization=False)
    #ret = rknn.build(do_quantization=True,dataset='./dataset.txt')
    if ret != 0:
        print('Build model failed!')
        exit(ret)
    print('done')

    # Export rknn model
    print('--> Export rknn model')
    ret = rknn.export_rknn('./model.rknn')
    if ret != 0:
        print('Export rknn model failed!')
        exit(ret)
    print('done')

#Init runtime environment
print('--> Init runtime environment')
ret = rknn.init_runtime()
#    if ret != 0:
#        print('Init runtime environment failed!')
#        exit(ret)
print('done')

img = cv2.imread(IMG_PATH)
img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
img = cv2.resize(img,(180,180))
img = np.expand_dims(img, 0)

#print('--> Accuracy analysis')
#rknn.accuracy_analysis(inputs=['./test.jpg'])
#print('done')

print('--> Running model')
outputs = rknn.inference(inputs=[img])
print(outputs)
outputs = tf.nn.softmax(outputs)
print(outputs)

print(
    "This image most likely belongs to {} with a {:.2f} percent confidence."
    .format(class_names[np.argmax(outputs)], 100 * np.max(outputs))
)
#print("The image prediction is:", class_names[np.argmax(outputs)])
print('done')

rknn.release()

Test results show:

W __init__: rknn-toolkit2 version: 1.4.0-22dcfef4
--> Config model
done
--> Loading model
done
--> Building model
done
--> Export rknn model
done
--> Init runtime environment
W init_runtime: Target is None, use simulator!
done
--> Running model
Analysing : 100%|███████████████████████████████████████████████████| 18/18 [00:00<00:00, 90.12it/s]
Preparing : 100%|███████████████████████████████████████████████████| 18/18 [00:01<00:00, 13.98it/s]
[array([[-4.5390625 , -1.2275391 , -0.47338867,  4.75      ,  0.34350586]],
  dtype=float32)]
tf.Tensor([[[9.0598274e-05 2.4848275e-03 5.2822586e-03 9.8018610e-01 1.1956180e-02]]], shape=(1, 1, 5), dtype=float32)
This image most likely belongs to sunflowers with a 98.02 percent confidence.
done

5.3. Deployment reasoning test

Deploy reasoning on the lubancat board, use RKNN-Toolkit-Lite2, take RK356X as an example:

rknnlite_inference.py(Reference supporting example)
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
IMG_PATH = 'test1.jpg'
RKNN_MODEL = 'model.rknn'
img_height = 180
img_width = 180
class_names = ['daisy', 'dandelion', 'roses', 'sunflowers', 'tulips']

# Create RKNN object
rknn_lite = RKNNLite()

# load RKNN model
print('--> Load RKNN model')
ret = rknn_lite.load_rknn(RKNN_MODEL)
if ret != 0:
    print('Load RKNN model failed')
    exit(ret)
print('done')

# Init runtime environment
print('--> Init runtime environment')
ret = rknn_lite.init_runtime()
if ret != 0:
    print('Init runtime environment failed!')
    exit(ret)
print('done')

# load image
img = cv2.imread(IMG_PATH)
img = cv2.resize(img,(180,180))
img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
img = np.expand_dims(img, 0)

# runing model
print('--> Running model')
outputs = rknn_lite.inference(inputs=[img])
print("result: ", outputs)
print(
    "This image most likely belongs to {}."
    .format(class_names[np.argmax(outputs)])
)

rknn_lite.release()

Simple test results on the board:

--> Load RKNN model
done
--> Init runtime environment
I RKNN: [17:28:01.594] RKNN Runtime Information: librknnrt version: 1.4.0 (a10f100eb@2022-09-09T09:07:14)
I RKNN: [17:28:01.594] RKNN Driver Information: version: 0.7.2
I RKNN: [17:28:01.595] RKNN Model Information: version: 1, toolkit version: 1.4.0-22dcfef4(compiler version: 1.4.0 (3b4520e4f@2022-09-05T20:52:35)), target: RKNPU lite, target platform: rk3568, framework name: TFLite, framework layout: NHWC
done
--> Running model
result:  [array([[-3.7226562, -1.2607422, -0.5805664,  3.5585938,  0.296875 ]],
    dtype=float32)]
This image most likely belongs to sunflowers.
done

5.4. Summarize

Use TensorFlow to complete a simple flower image classification, and convert the model to the RKNN model. It is deployed on the Lubancat board. The model verification accuracy is more than 70%, and it is used for simple learning. This model test comes from the image classification tutorial <https://tensorflow.google.cn/tutorials/classification> _.