1. 前言Qwen2-VL是一个基于视觉-语言预训练的多模态模型支持图像和文本的联合输入输出是文本形式。将在正点原子RK3588板卡上部署Qwen2-VL-2B-Instruct模型对输入图像进行描述。2. 版本2.1 kernel版本本次部署的kernel版本rootATK-DLRK3588-Ubuntu:~# uname -a Linux ATK-DLRK3588-Ubuntu 5.10.160 #2 SMP Mon Apr 14 21:43:53 CST 2025 aarch64 aarch64 aarch64 GNU/Linux2.2 ubuntu 版本rootATK-DLRK3588-Ubuntu:~# cat /etc/issue Ubuntu 20.04.6 LTS \n \l2.3 npu2.3.1 npu 驱动版本rootATK-DLRK3588-Ubuntu:~# cat /sys/kernel/debug/rknpu/version RKNPUdriver:v0.9.82.3.2 编译0.9.8驱动如何编译最新的驱动以及解决编译过程中遇到的两个问题可以参考这篇博文rk3588 npu升级修订编译正点原子的kernel参考正点原子的官方文档替换npu驱动并从新编译2.3.3 升级kernel将更新最新rknpu驱动的kernel升级。RK可以单独升级kernel。截图如下2.3.4 rknn-llmrknn-llm 提供大模型转换和和板端推理功能。本次PC端和板端使用的版本一致版本如下2.3.5 rknn-toolkit2rknn-toolkit2提供通用的模型转换和板端推理功能。本次使用的版本如下2.4 说明最好保证版本的一致除了kernel需要编译最新版本的npu驱动 0.9.8之外其他的全部使用正点原子原厂提供的即可3. 模型转换3.1 已经转换好的模型通过网盘分享的文件Qwen2-VL-2B链接:https://pan.baidu.com/s/1CBEoRM2bW5zoTsXWNRk1dw?pwdij5d提取码:ij5d3.2 模型转换模型转换有非常多得参考文档主要参考一下如下RKLLM 部署文档Qwen2-VL 模型转换文档本人也是参考以上两篇文档完成RKLLM部署和模型转换的。4. Qwen2-VL 板端推理4.1 模型拷贝将完成转换的模型拷贝到设备4.2 rknn-llm 推理代码修订下载 rknn-llm到设备修订build-linux.sh文件rm -rf build mkdir build cd build cmake .. -DCMAKE_CXX_COMPILER/usr/bin/aarch64-linux-gnu-g \ -DCMAKE_C_COMPILER/usr/bin/aarch64-linux-gnu-gcc \ -DCMAKE_BUILD_TYPERelease \ -DCMAKE_SYSTEM_NAMELinux \ -DCMAKE_SYSTEM_PROCESSORaarch64 \ make -j8 make install4.3 rknn-llm 推理代码编译修改编译脚本之后进入到examples/Qwen2-VL-2B_Demo/deploy 路径之后执行rootATK-DLRK3588-Ubuntu:/work/rknn-llm/examples/Qwen2-VL-2B_Demo/deploy# ./build-linux.sh -- The C compiler identification is GNU 9.4.0 -- The CXX compiler identification is GNU 9.4.0 -- Check for working Ccompiler:/usr/bin/aarch64-linux-gnu-gcc -- Check for working Ccompiler:/usr/bin/aarch64-linux-gnu-gcc -- works -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Detecting C compile features -- Detecting C compile features - done -- Check for working CXXcompiler:/usr/bin/aarch64-linux-gnu-g -- Check for working CXXcompiler:/usr/bin/aarch64-linux-gnu-g -- works -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Detecting CXX compile features -- Detecting CXX compile features - done -- FoundOpenCV:/work/rknn-llm/examples/Qwen2-VL-2B_Demo/deploy/3rdparty/opencv-linux-aarch64(found version3.4.5)-- Configuring done -- Generating done -- Build files have been writtento:/work/rknn-llm/examples/Qwen2-VL-2B_Demo/deploy/build Scanning dependencies of target demo Scanning dependencies of target llm Scanning dependencies of target imgenc [ 12%] Building CXX object CMakeFiles/llm.dir/src/llm.cpp.occ1plus:warning:/work/rknn-llm/examples/Qwen2-VL-2B_Demo/deploy/src/image_enc.h:not a directory [ 25%] Building CXX object CMakeFiles/demo.dir/src/image_enc.cc.o [ 37%] Building CXX object CMakeFiles/imgenc.dir/src/image_enc.cc.o [ 62%] Building CXX object CMakeFiles/imgenc.dir/src/img_encoder.cpp.o [ 62%] Building CXX object CMakeFiles/demo.dir/src/main.cpp.occ1plus:warning:/work/rknn-llm/examples/Qwen2-VL-2B_Demo/deploy/src/image_enc.h:not a directorycc1plus:warning:/work/rknn-llm/examples/Qwen2-VL-2B_Demo/deploy/src/image_enc.h:not a directorycc1plus:warning:/work/rknn-llm/examples/Qwen2-VL-2B_Demo/deploy/src/image_enc.h:not a directorycc1plus:warning:/work/rknn-llm/examples/Qwen2-VL-2B_Demo/deploy/src/image_enc.h:not a directory [ 75%] Linking CXX executable llm [ 75%] Built target llm [ 87%] Linking CXX executable imgenc [100%] Linking CXX executable demo [100%] Built target demo [100%] Built target imgenc [ 37%] Built target demo [ 62%] Built target llm [100%] Built target imgenc Install the project... -- Installconfiguration:Release--Installing:/work/rknn-llm/examples/Qwen2-VL-2B_Demo/deploy/install/demo_Linux_aarch64/./imgenc -- Set runtime path of/work/rknn-llm/examples/Qwen2-VL-2B_Demo/deploy/install/demo_Linux_aarch64/./imgencto--Installing:/work/rknn-llm/examples/Qwen2-VL-2B_Demo/deploy/install/demo_Linux_aarch64/./llm -- Set runtime path of/work/rknn-llm/examples/Qwen2-VL-2B_Demo/deploy/install/demo_Linux_aarch64/./llmto--Installing:/work/rknn-llm/examples/Qwen2-VL-2B_Demo/deploy/install/demo_Linux_aarch64/./demo -- Set runtime path of/work/rknn-llm/examples/Qwen2-VL-2B_Demo/deploy/install/demo_Linux_aarch64/./demoto--Up-to-date:/work/rknn-llm/examples/Qwen2-VL-2B_Demo/deploy/install/demo_Linux_aarch64/lib/librknnrt.so --Up-to-date:/work/rknn-llm/examples/Qwen2-VL-2B_Demo/deploy/install/demo_Linux_aarch64/lib/librkllmrt.so --Up-to-date:/work/rknn-llm/examples/Qwen2-VL-2B_Demo/deploy/install/demo_Linux_aarch64/./demo.jpg rootATK-DLRK3588-Ubuntu:/work/rknn-llm/examples/Qwen2-VL-2B_Demo/deploy#4.4 验证4.4.1 注意事项注意需要导出 librkllmrt.so 库的路径否则会提示错误./demo:error while loading sharedlibraries:librkllmrt.so:cannot open shared objectfile:No such file or directory导出库export LD_LIBRARY_PATH./lib4.4.2 验证执行代码rootATK-DLRK3588-Ubuntu:/work/rknn-llm/examples/Qwen2-VL-2B_Demo/deploy/install/demo_Linux_aarch64# export LD_LIBRARY_PATH./lib rootATK-DLRK3588-Ubuntu:/work/rknn-llm/examples/Qwen2-VL-2B_Demo/deploy/install/demo_Linux_aarch64# ./demo demo.jpg /work/qianwen/qwen2_vl_2b_vision_rk3588.rknn /work/qianwen/Qwen2-VL-2B-Instruct.rkllm 128 512验证结果rootATK-DLRK3588-Ubuntu:/work/rknn-llm/examples/Qwen2-VL-2B_Demo/deploy/install/demo_Linux_aarch64# export LD_LIBRARY_PATH./lib rootATK-DLRK3588-Ubuntu:/work/rknn-llm/examples/Qwen2-VL-2B_Demo/deploy/install/demo_Linux_aarch64# ./demo demo.jpg /work/qianwen/qwen2_vl_2b_vision_rk3588.rknn /work/qianwen/Qwen2-VL-2B-Instruct.rkllm 128 512 Irkllm:rkllm-runtimeversion:1.2.0,rknpu driverversion:0.9.8,platform:RK3588 Irkllm:loading rkllm model from /work/qianwen/Qwen2-VL-2B-Instruct.rkllm Irkllm:rkllm-toolkitversion:1.2.0,max_context_limit:4096,npu_core_num:3,target_platform:RK3588 Irkllm:Enabledcpus:[4,5,6,7] Irkllm:Enabled cpusnum:4 Irkllm:system_prompt:|im_start|system\nYou are a helpful assistant.|im_end|\n Irkllm:prompt_prefix:|im_start|user\n Irkllm:prompt_postfix:|im_end|\n|im_start|assistant\n rkllm init successmain:LLM Model loaded in 8587.09 ms model inputnum:1,outputnum:1 inputtensors:index0,nameonnx::Expand_0,n_dims4,dims[1,392,392,3],n_elems460992,size921984,fmtNHWC,typeFP16,qnt_typeAFFINE,zp0,scale1.000000 outputtensors:index0,name6460,n_dims2,dims[196,1536,0,0],n_elems301056,size602112,fmtUNDEFINED,typeFP16,qnt_typeAFFINE,zp0,scale1.000000 model input height392,width392,channel3main:ImgEnc Model loaded in 6957.11 ms Irkllm:reset chattemplate:Irkllm:system_prompt:|im_start|system\nYou are a helpful assistant.|im_end|\n Irkllm:prompt_prefix:|im_start|user\n Irkllm:prompt_postfix:|im_end|\n|im_start|assistant\n **********************可输入以下问题对应序号获取回答/或自定义输入******************** [0] imageWhat is in the image? [1] image这张图片中有什么 *************************************************************************user:1 image这张图片中有什么robot:这张图片展示了一位穿着宇航服的宇航员他坐在月球表面的一块岩石上。背景是地球和星空显得非常科幻和未来感。宇航员手中拿着一个绿色的瓶子似乎在庆祝或享受某种活动。整体画面充满了神秘和探索的感觉。user:
极物科技 正点原子 RK3588 部署 Qwen2-VL Qwen2-VL-2B-Instruct，提供模型和npu驱动0.9.8的kernel
1. 前言Qwen2-VL是一个基于视觉-语言预训练的多模态模型支持图像和文本的联合输入输出是文本形式。将在正点原子RK3588板卡上部署Qwen2-VL-2B-Instruct模型对输入图像进行描述。2. 版本2.1 kernel版本本次部署的kernel版本rootATK-DLRK3588-Ubuntu:~# uname -a Linux ATK-DLRK3588-Ubuntu 5.10.160 #2 SMP Mon Apr 14 21:43:53 CST 2025 aarch64 aarch64 aarch64 GNU/Linux2.2 ubuntu 版本rootATK-DLRK3588-Ubuntu:~# cat /etc/issue Ubuntu 20.04.6 LTS \n \l2.3 npu2.3.1 npu 驱动版本rootATK-DLRK3588-Ubuntu:~# cat /sys/kernel/debug/rknpu/version RKNPUdriver:v0.9.82.3.2 编译0.9.8驱动如何编译最新的驱动以及解决编译过程中遇到的两个问题可以参考这篇博文rk3588 npu升级修订编译正点原子的kernel参考正点原子的官方文档替换npu驱动并从新编译2.3.3 升级kernel将更新最新rknpu驱动的kernel升级。RK可以单独升级kernel。截图如下2.3.4 rknn-llmrknn-llm 提供大模型转换和和板端推理功能。本次PC端和板端使用的版本一致版本如下2.3.5 rknn-toolkit2rknn-toolkit2提供通用的模型转换和板端推理功能。本次使用的版本如下2.4 说明最好保证版本的一致除了kernel需要编译最新版本的npu驱动 0.9.8之外其他的全部使用正点原子原厂提供的即可3. 模型转换3.1 已经转换好的模型通过网盘分享的文件Qwen2-VL-2B链接:https://pan.baidu.com/s/1CBEoRM2bW5zoTsXWNRk1dw?pwdij5d提取码:ij5d3.2 模型转换模型转换有非常多得参考文档主要参考一下如下RKLLM 部署文档Qwen2-VL 模型转换文档本人也是参考以上两篇文档完成RKLLM部署和模型转换的。4. Qwen2-VL 板端推理4.1 模型拷贝将完成转换的模型拷贝到设备4.2 rknn-llm 推理代码修订下载 rknn-llm到设备修订build-linux.sh文件rm -rf build mkdir build cd build cmake .. -DCMAKE_CXX_COMPILER/usr/bin/aarch64-linux-gnu-g \ -DCMAKE_C_COMPILER/usr/bin/aarch64-linux-gnu-gcc \ -DCMAKE_BUILD_TYPERelease \ -DCMAKE_SYSTEM_NAMELinux \ -DCMAKE_SYSTEM_PROCESSORaarch64 \ make -j8 make install4.3 rknn-llm 推理代码编译修改编译脚本之后进入到examples/Qwen2-VL-2B_Demo/deploy 路径之后执行rootATK-DLRK3588-Ubuntu:/work/rknn-llm/examples/Qwen2-VL-2B_Demo/deploy# ./build-linux.sh -- The C compiler identification is GNU 9.4.0 -- The CXX compiler identification is GNU 9.4.0 -- Check for working Ccompiler:/usr/bin/aarch64-linux-gnu-gcc -- Check for working Ccompiler:/usr/bin/aarch64-linux-gnu-gcc -- works -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Detecting C compile features -- Detecting C compile features - done -- Check for working CXXcompiler:/usr/bin/aarch64-linux-gnu-g -- Check for working CXXcompiler:/usr/bin/aarch64-linux-gnu-g -- works -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Detecting CXX compile features -- Detecting CXX compile features - done -- FoundOpenCV:/work/rknn-llm/examples/Qwen2-VL-2B_Demo/deploy/3rdparty/opencv-linux-aarch64(found version3.4.5)-- Configuring done -- Generating done -- Build files have been writtento:/work/rknn-llm/examples/Qwen2-VL-2B_Demo/deploy/build Scanning dependencies of target demo Scanning dependencies of target llm Scanning dependencies of target imgenc [ 12%] Building CXX object CMakeFiles/llm.dir/src/llm.cpp.occ1plus:warning:/work/rknn-llm/examples/Qwen2-VL-2B_Demo/deploy/src/image_enc.h:not a directory [ 25%] Building CXX object CMakeFiles/demo.dir/src/image_enc.cc.o [ 37%] Building CXX object CMakeFiles/imgenc.dir/src/image_enc.cc.o [ 62%] Building CXX object CMakeFiles/imgenc.dir/src/img_encoder.cpp.o [ 62%] Building CXX object CMakeFiles/demo.dir/src/main.cpp.occ1plus:warning:/work/rknn-llm/examples/Qwen2-VL-2B_Demo/deploy/src/image_enc.h:not a directorycc1plus:warning:/work/rknn-llm/examples/Qwen2-VL-2B_Demo/deploy/src/image_enc.h:not a directorycc1plus:warning:/work/rknn-llm/examples/Qwen2-VL-2B_Demo/deploy/src/image_enc.h:not a directorycc1plus:warning:/work/rknn-llm/examples/Qwen2-VL-2B_Demo/deploy/src/image_enc.h:not a directory [ 75%] Linking CXX executable llm [ 75%] Built target llm [ 87%] Linking CXX executable imgenc [100%] Linking CXX executable demo [100%] Built target demo [100%] Built target imgenc [ 37%] Built target demo [ 62%] Built target llm [100%] Built target imgenc Install the project... -- Installconfiguration:Release--Installing:/work/rknn-llm/examples/Qwen2-VL-2B_Demo/deploy/install/demo_Linux_aarch64/./imgenc -- Set runtime path of/work/rknn-llm/examples/Qwen2-VL-2B_Demo/deploy/install/demo_Linux_aarch64/./imgencto--Installing:/work/rknn-llm/examples/Qwen2-VL-2B_Demo/deploy/install/demo_Linux_aarch64/./llm -- Set runtime path of/work/rknn-llm/examples/Qwen2-VL-2B_Demo/deploy/install/demo_Linux_aarch64/./llmto--Installing:/work/rknn-llm/examples/Qwen2-VL-2B_Demo/deploy/install/demo_Linux_aarch64/./demo -- Set runtime path of/work/rknn-llm/examples/Qwen2-VL-2B_Demo/deploy/install/demo_Linux_aarch64/./demoto--Up-to-date:/work/rknn-llm/examples/Qwen2-VL-2B_Demo/deploy/install/demo_Linux_aarch64/lib/librknnrt.so --Up-to-date:/work/rknn-llm/examples/Qwen2-VL-2B_Demo/deploy/install/demo_Linux_aarch64/lib/librkllmrt.so --Up-to-date:/work/rknn-llm/examples/Qwen2-VL-2B_Demo/deploy/install/demo_Linux_aarch64/./demo.jpg rootATK-DLRK3588-Ubuntu:/work/rknn-llm/examples/Qwen2-VL-2B_Demo/deploy#4.4 验证4.4.1 注意事项注意需要导出 librkllmrt.so 库的路径否则会提示错误./demo:error while loading sharedlibraries:librkllmrt.so:cannot open shared objectfile:No such file or directory导出库export LD_LIBRARY_PATH./lib4.4.2 验证执行代码rootATK-DLRK3588-Ubuntu:/work/rknn-llm/examples/Qwen2-VL-2B_Demo/deploy/install/demo_Linux_aarch64# export LD_LIBRARY_PATH./lib rootATK-DLRK3588-Ubuntu:/work/rknn-llm/examples/Qwen2-VL-2B_Demo/deploy/install/demo_Linux_aarch64# ./demo demo.jpg /work/qianwen/qwen2_vl_2b_vision_rk3588.rknn /work/qianwen/Qwen2-VL-2B-Instruct.rkllm 128 512验证结果rootATK-DLRK3588-Ubuntu:/work/rknn-llm/examples/Qwen2-VL-2B_Demo/deploy/install/demo_Linux_aarch64# export LD_LIBRARY_PATH./lib rootATK-DLRK3588-Ubuntu:/work/rknn-llm/examples/Qwen2-VL-2B_Demo/deploy/install/demo_Linux_aarch64# ./demo demo.jpg /work/qianwen/qwen2_vl_2b_vision_rk3588.rknn /work/qianwen/Qwen2-VL-2B-Instruct.rkllm 128 512 Irkllm:rkllm-runtimeversion:1.2.0,rknpu driverversion:0.9.8,platform:RK3588 Irkllm:loading rkllm model from /work/qianwen/Qwen2-VL-2B-Instruct.rkllm Irkllm:rkllm-toolkitversion:1.2.0,max_context_limit:4096,npu_core_num:3,target_platform:RK3588 Irkllm:Enabledcpus:[4,5,6,7] Irkllm:Enabled cpusnum:4 Irkllm:system_prompt:|im_start|system\nYou are a helpful assistant.|im_end|\n Irkllm:prompt_prefix:|im_start|user\n Irkllm:prompt_postfix:|im_end|\n|im_start|assistant\n rkllm init successmain:LLM Model loaded in 8587.09 ms model inputnum:1,outputnum:1 inputtensors:index0,nameonnx::Expand_0,n_dims4,dims[1,392,392,3],n_elems460992,size921984,fmtNHWC,typeFP16,qnt_typeAFFINE,zp0,scale1.000000 outputtensors:index0,name6460,n_dims2,dims[196,1536,0,0],n_elems301056,size602112,fmtUNDEFINED,typeFP16,qnt_typeAFFINE,zp0,scale1.000000 model input height392,width392,channel3main:ImgEnc Model loaded in 6957.11 ms Irkllm:reset chattemplate:Irkllm:system_prompt:|im_start|system\nYou are a helpful assistant.|im_end|\n Irkllm:prompt_prefix:|im_start|user\n Irkllm:prompt_postfix:|im_end|\n|im_start|assistant\n **********************可输入以下问题对应序号获取回答/或自定义输入******************** [0] imageWhat is in the image? [1] image这张图片中有什么 *************************************************************************user:1 image这张图片中有什么robot:这张图片展示了一位穿着宇航服的宇航员他坐在月球表面的一块岩石上。背景是地球和星空显得非常科幻和未来感。宇航员手中拿着一个绿色的瓶子似乎在庆祝或享受某种活动。整体画面充满了神秘和探索的感觉。user:
