esp32-s3训练自己的数据进行目标检测、图像分类

esp-idf的安装参考我前面的文章： esp32cam和esp32-s3烧录human_face_detect实现人脸识别

一、下载项目

• 训练、转换模型：ModelAssistant(main)
• 部署模型：sscma-example-esp32(1.0.0)
• 说明文档：sscma-model-zoo

二、环境

python3.8 + CUDA11.7 + esp-idf5.0
# 主要按照ModelAssistant/requirements_cuda.txt，如果训练时有库不兼容的问题可参考下方
torch                        2.0.0+cu117
torchaudio                   2.0.1+cu117
torchvision                  0.15.1+cu117
yapf                         0.40.2
typing_extensions            4.5.0
tensorboard                  2.13.0
tensorboard-data-server      0.7.2
tensorflow                   2.13.0
keras                        2.13.1
tensorflow-estimator         2.13.0
tensorflow-intel             2.13.0
tensorflow-io-gcs-filesystem 0.31.0
sscma                        2.0.0rc3
setuptools                   60.2.0
rich                         13.4.2
Pillow                       9.4.0
mmcls                        1.0.0rc6
mmcv                         2.0.0
mmdet                        3.0.0
mmengine                     0.10.1
mmpose                       1.2.0
mmyolo                       0.5.0

三、训练和导出模型

• step 1: 将voc格式的标注文件转换为edgelab的训练格式，并按8：2的比例划分为训练集和验证集

import os
import json
import pandas as pd
from xml.etree import ElementTree as ET
from PIL import Image
import shutil
import random
from tqdm import tqdm# Set paths
voc_path = 'F:/datasets/VOCdevkit/VOC2007'
train_path = 'F:/edgelab/ModelAssistant/datasets/myself/train'
valid_path = 'F:/edgelab/ModelAssistant/datasets/meself/valid'# 只读取有目标的，且属于需要训练的类别
classes = ["face"]# Create directories if not exist
if not os.path.exists(train_path):os.makedirs(train_path)
if not os.path.exists(valid_path):os.makedirs(valid_path)# Get list of image files
image_files = os.listdir(os.path.join(voc_path, 'JPEGImages'))
random.seed(0)
random.shuffle(image_files)# Split data into train and valid
train_files = image_files[:int(len(image_files)*0.8)]
valid_files = image_files[int(len(image_files)*0.8):]# Convert train data to COCO format
train_data = {'categories': [], 'images': [], 'annotations': []}
train_ann_id = 0
train_cat_id = 0
img_id = 0
train_categories = {}
for file in tqdm(train_files):# Add annotationsxml_file = os.path.join(voc_path, 'Annotations', file[:-4] + '.xml')tree = ET.parse(xml_file)root = tree.getroot()for obj in root.findall('object'):category = obj.find('name').textif category not in classes:continueif category not in train_categories:train_categories[category] = train_cat_idtrain_cat_id += 1category_id = train_categories[category]bbox = obj.find('bndbox')x1 = int(bbox.find('xmin').text)y1 = int(bbox.find('ymin').text)x2 = int(bbox.find('xmax').text)y2 = int(bbox.find('ymax').text)width = x2 - x1height = y2 - y1ann_info = {'id': train_ann_id, 'image_id': img_id, 'category_id': category_id, 'bbox': [x1, y1, width, height],'area': width*height, 'iscrowd': 0}train_data['annotations'].append(ann_info)train_ann_id += 1if len(root.findall('object')):# 只有有目标的图片才加进来image_id = img_idimg_id += 1image_file = os.path.join(voc_path, 'JPEGImages', file)shutil.copy(image_file, os.path.join(train_path, file))img = Image.open(image_file)image_info = {'id': image_id, 'file_name': file, 'width': img.size[0], 'height': img.size[1]}train_data['images'].append(image_info)# Add categories
for category, category_id in train_categories.items():train_data['categories'].append({'id': category_id, 'name': category})# Save train data to file
with open(os.path.join(train_path, '_annotations.coco.json'), 'w') as f:json.dump(train_data, f, indent=4)# Convert valid data to COCO format
valid_data = {'categories': [], 'images': [], 'annotations': []}
valid_ann_id = 0
img_id = 0
for file in tqdm(valid_files):# Add annotationsxml_file = os.path.join(voc_path, 'Annotations', file[:-4] + '.xml')tree = ET.parse(xml_file)root = tree.getroot()for obj in root.findall('object'):category = obj.find('name').textif category not in classes:continuecategory_id = train_categories[category]bbox = obj.find('bndbox')x1 = int(bbox.find('xmin').text)y1 = int(bbox.find('ymin').text)x2 = int(bbox.find('xmax').text)y2 = int(bbox.find('ymax').text)width = x2 - x1height = y2 - y1ann_info = {'id': valid_ann_id, 'image_id': img_id, 'category_id': category_id, 'bbox': [x1, y1, width, height],'area': width*height, 'iscrowd': 0}valid_data['annotations'].append(ann_info)valid_ann_id += 1if len(root.findall('object')):# Add imageimage_id = img_idimg_id += 1image_file = os.path.join(voc_path, 'JPEGImages', file)shutil.copy(image_file, os.path.join(valid_path, file))img = Image.open(image_file)image_info = {'id': image_id, 'file_name': file, 'width': img.size[0], 'height': img.size[1]}valid_data['images'].append(image_info)# Add categories
valid_data['categories'] = train_data['categories']# Save valid data to file
with open(os.path.join(valid_path, '_annotations.coco.json'), 'w') as f:json.dump(valid_data, f, indent=4)

• step 2: 参考Face Detection - Swift-YOLO下载模型权重文件和训练

python tools/train.py configs/yolov5/yolov5_tiny_1xb16_300e_coco.py \
--cfg-options  \work_dir=work_dirs/face_96 \num_classes=3 \epochs=300  \height=96 \width=96 \batch=128 \data_root=datasets/face/ \load_from=datasets/face/pretrain.pth

• step 3: 训练过程可视化tensorboard

cd work_dirs/face_96/20231219_181418/vis_data
tensorboard --logdir=./

然后按照提示打开http://localhost:6006/

• step 4: 导出模型

python tools/export.py configs/yolov5/yolov5_tiny_1xb16_300e_coco.py ./work_dirs/face_96/best_coco_bbox_mAP_epoch_300.pth --target tflite onnx
--cfg-options  \work_dir=work_dirs/face_96 \num_classes=3 \epochs=300  \height=96 \width=96 \batch=128 \data_root=datasets/face/ \load_from=datasets/face/pretrain.pth

这样就会在./work_dirs/face_96路径下生成best_coco_bbox_mAP_epoch_300_int8.tflite文件了。

四、部署模型

• step 1: 将best_coco_bbox_mAP_epoch_300_int8.tflite复制到F:\edgelab\sscma-example-esp32-1.0.0\model_zoo路径下
• step 2: 参照edgelab-example-esp32-训练和部署一个FOMO模型将模型转换为C语言文件，并将其放入到F:\edgelab\sscma-example-esp32-1.0.0\components\modules\model路径下

python tools/tflite2c.py --input ./model_zoo/best_coco_bbox_mAP_epoch_300_int8.tflite --name yolo --output_dir ./components/modules/model --classes face

这样会生成./components/modules/model/yolo_model_data.cpp和yolo_model_data.h两个文件。

• step 3: 利用idf烧录程序
  

fb_gfx_printf(frame, yolo.x - yolo.w / 2, yolo.y - yolo.h/2 - 5, 0x1FE0, "%s:%d", g_yolo_model_classes[yolo.target], yolo.confidence);

打开esp-idf cmd

cd F:\edgelab\sscma-example-esp32-1.0.0\examples\yolo
idf.py set-target esp32s3
idf.py menuconfig

勾选上方的这个选项不然报错

E:/Softwares/Espressif/frameworks/esp-idf-v5.0.4/components/driver/deprecated/driver/i2s.h:27:2: warning: #warning "This set of I2S APIs has been deprecated, please include 'driver/i2s_std.h', 'driver/i2s_pdm.h' or 'driver/i2s_tdm.h' instead. if you want to keep using the old APIs and ignore this warning, you can enable 'Suppress leagcy driver deprecated warning' option under 'I2S Configuration' menu in Kconfig" [-Wcpp]27 | #warning "This set of I2S APIs has been deprecated, \|  ^~~~~~~
ninja: build stopped: subcommand failed.
ninja failed with exit code 1, output of the command is in the F:\edgelab\sscma-example-esp32-1.0.0\examples\yolo\build\log\idf_py_stderr_output_27512 and F:\edgelab\sscma-example-esp32-1.0.0\examples\yolo\build\log\idf_py_stdout_output_27512

idf.py flash monitor -p COM3

lcd端也能实时显示识别结果，输入大小为96x96时推理时间大概200ms，192x192时时间大概660ms

五、存在的问题

该链路中量化是比较简单的，在我的数据集上量化后精度大打折扣，应该需要修改量化算法，后续再说吧。

• 量化前
  
• 量化后