CLIP（Contrastive Language-Image Pre-Training）在SOPHON BM1684X上进行推理

1、链接

https://github.com/sophgo/sophon-demo/tree/release/sample/CLIP

2、开发环境中交叉编译生成sophon_arm-3.8.0-py3-none-any.whl

3、sail安装

算能官网技术资料中SDK-24.04.01的
libsophon_soc_0.4.1_aarch64.tar.gz
sophon-mw-soc_0.4.1_aarch64.tar.gz
SOPHON-SAIL源码

4、对应的python版本

python3 -m dfss --url=[e-mail]:/toolchains/pythons/Python-3.8.2.tar.gz

5、交叉编译sail源码

sail目录下mkdir build && cd buildcmake -DBUILD_TYPE=soc  \-DCMAKE_TOOLCHAIN_FILE=../cmake/BM168x_SOC/ToolChain_aarch64_linux.cmake \-DPYTHON_EXECUTABLE=./python_3.8.2/bin/python3 \-DCUSTOM_PY_LIBDIR=./python_3.8.2/lib \-DLIBSOPHON_BASIC_PATH=/sdk/libsophon_0.5.1_aarch64/opt/sophon/libsophon-0.5.1 \-DFFMPEG_BASIC_PATH=/sdk/sophon-mw_0.10.0_aarch64/opt/sophon/sophon-ffmpeg_0.10.0 \-DOPENCV_BASIC_PATH=/sdk/sophon-mw_0.10.0_aarch64/opt/sophon/sophon-opencv_0.10.0 ..make pysail打包生成whl包cd ../python/socchmod +x sophon_soc_whl.sh./sophon_soc_whl.sh

6、BM1684X安装环境

sudo apt install unzip
sudo apt install p7zip p7zip-fullpip3 install ftfy
pip3 install regex
pip3 install torch
pip3 install torchvision
pip3 install --upgrade torch --no-cache-dir

pip3 install pillow
pip3 install streamlit
pip3 install certifi
pip3 install cryptography
pip3 install pyOpenSSL

安装编译生成的sophon_arm-3.8.0-py3-none-any.whl
pip3 install sophon_arm-3.8.0-py3-none-any.whl --force-reinstall

7、启动CLIP后端 前端

export PYTHONPATH=$PYTHONPATH:/opt/sophon/sophon-opencv_0.10.0/opencv-python
python3 server-backend.py --host 0.0.0.0 --port 8080 
streamlit run server-front.py "http://localhost:8080" --server.address 0.0.0.0 --server.port 5000 

8、浏览器访问BM1684X启动CLIP的IP地址
http://IP地址:5000/