CANN/asnumpy开发指南

CANN/asnumpy开发指南 Asnumpy 开发指南【免费下载链接】asnumpy哈尔滨工业大学计算学部苏统华、王甜甜老师团队联合华为CANN团队开发的华为昇腾NPU原生Numpy仓库项目地址: https://gitcode.com/cann/asnumpy目录项目概述开发流程Branch Management后端开发 (C)绑定层 (Pybind11)前端开发 (Python)测试编写编译与运行一、项目概述Asnumpy 是一个基于华为昇腾 NPU 的数值计算库提供与 NumPy 兼容的 API将计算任务转移到 NPU 上执行。项目采用分层架构后端层 (C): 使用昇腾 CANN API 实现 NPU 算子绑定层 (Pybind11): 将 C 函数绑定到 Python 接口前端层 (Python): 提供用户友好的 API与 NumPy 接口保持一致测试层: 使用 pytest 框架和自定义测试工具验证功能正确性项目目录结构asnumpy/ ├── asnumpy/ # Python 包 │ ├── __init__.py # 主包初始化 │ ├── math.py # 数学模块 │ ├── linalg/ # 线性代数模块 │ ├── random/ # 随机数模块 │ └── ... ├── include/ # C 头文件 │ └── asnumpy/ │ ├── math/ # 数学模块头文件 │ └── ... ├── src/ # C 源文件 │ ├── math/ # 数学模块实现 │ └── ... ├── python/ # Pybind11 绑定 │ ├── bind_math.cpp # 数学模块绑定 │ └── ... ├── tests/ # 测试文件 │ └── asnumpy_tests/ # 测试用例 └── ...二、开发流程开发新功能通常遵循以下步骤1. 添加函数声明 (include/) ↓ 2. 实现函数逻辑 (src/) ↓ 3. 添加 Python 绑定 (python/) ↓ 4. 添加 Python 包装层 (asnumpy/math.py) ↓ 5. 导出到主命名空间 (asnumpy/__init__.py) ↓ 6. 编写测试用例 (tests/) ↓ 7. 编译并运行测试三、Branch Management3.1 OverviewAsnumpy uses a simplified branching model based on a single main branch with release branches for stabilization:master ─── PR_A ─── PR_B ─── PR_C ─── PR_D ─── PR_E ─── PR_F │ │ release/v0.2.0 cherry-pick hotfix │ │ tag v0.2.0 tag v0.2.1 tag v0.2.1 │ PR to master3.2 Branch TypesBranchPurposeLifetimemasterMain branch. All feature PRs merge here.Permanentfeature/nameDevelop a new feature or enhancement.Short-livedfix/nameFix a bug.Short-livedrelease/vX.Y.ZStabilize and test a release.Per releasehotfix/nameBackport a fix from a release branch to master.Short-lived3.3 WorkflowFeature DevelopmentCreate a feature branch frommaster:git checkout master git pull upstream master git checkout -b feature/my-featureDevelop, test, and commit.Push and open a Pull Request targetingmaster.Ensure code review and CI pass.Merge the PR.Release ProcessCut a release branch frommaster:git checkout master git checkout -b release/v0.3.0Run tests and stabilize. Fix any issues directly on the release branch.Tag the release:git tag v0.3.0If additional fixes are needed after tagging, apply them on the release branch and tag a new patch version (e.g.,v0.3.1).Cherry-pick those fixes back tomaster(see Hotfix Process).Delete the release branch when the release series is no longer maintained:git branch -d release/v0.3.0Hotfix ProcessWhen a bug is fixed on a release branch, the fix must be brought back tomasterso that future releases include it:Fix the bug on the release branch and tag the patch release.Cherry-pick the fix to a new hotfix branch:git checkout -b hotfix/backport-fix-xxx master git cherry-pick commit-hash-of-the-fixPush the hotfix branch and open a Pull Request targetingmaster.This PR goes through the normal code review and CI pipeline.After merge, delete the hotfix branch:git branch -d hotfix/backport-fix-xxxWhy use a separate hotfix branch instead of cherry-picking directly to master?A dedicated branch allows the fix to go through PR review and CI checks, ensuring the same quality standards as any other contribution.3.4 Merge StrategyAlways use merge commits. Never squash.# When merging PRs on the platform, select Merge (not Squash and merge). # When merging locally: git merge --no-ff feature/my-featureRationale:Squash merges create a new commit that discards the original commit history and authorship. This breaks contribution statistics on the repository homepage and makes it harder to trace changes back to their original authors. Merge commits preserve the full history and ensure every contributor is properly credited.3.5 Branch Naming ConventionsPatternExampleDescriptionfeature/namefeature/add-sinc-functionNew feature or enhancementfix/namefix/incorrect-signbitBug fixrelease/vX.Y.Zrelease/v0.3.0Release stabilizationhotfix/namehotfix/backport-signbit-fixBackported fix from a release branchUse lowercase kebab-case for branch names. Keep names concise but descriptive.3.6 CI/CD PipelineCI/CD pipelines are triggered by events, not by branch types. Configure the following rules:EventPipelineDescriptionPR created/updated targetingmasterCIRun tests, linting, and code analysis on every PRPush tomaster(post-merge)CIVerify the merged code is healthyTag created (vX.Y.Z)CDBuild release artifacts, publish packagesNo additional CI configuration is needed forfeature/,fix/, orhotfix/branches. As long as the PR targetsmaster, CI will run regardless of the source branch name.3.7 Contributor WorkflowTeam Members (with push access)Team members create branches directly in the upstream repository:upstream/master ←── PR ─── upstream/fix/xxxgit clone upstream-url git checkout master git pull upstream master git checkout -b fix/signbit-error # Develop, commit, push git push upstream fix/signbit-error # Open a PR on the platform: fix/signbit-error → masterExternal Contributors (without push access)External contributors work in their own fork:upstream/master ←── PR ─── fork/fix/xxxgit clone fork-url git checkout master git pull upstream master git checkout -b fix/signbit-error # Develop, commit, push git push fork fix/signbit-error # Open a PR on the platform: fork/fix/xxx → upstream/masterKey PointsOne PR per change.Always create a PR directly from the working branch tomaster. There is no need for a two-step PR process (e.g., PR to a fix branch, then PR from fix branch to master).All code changes go through PR.No direct pushes tomaster.All PRs go through CI and code review.Regardless of whether the contributor is a team member or an external contributor.3.8 Commit Hygiene and Review NormsCommit QualityEvery commit in a PR should represent a single, meaningful change. Before submitting a PR, contributors are expected to clean up their commit history:Bad (fragmented commits that pad the count):fix typo fix another typo update import add sinc function add sinc testGood (clean, meaningful commits):feat: add sinc function test: add sinc unit testsCleaning Up Commits Before Submitting a PRUse interactive rebase to squash trivial commits:git rebase -i master # Mark trivial commits with s (squash) or f (fixup) # Keep only meaningful commits as p (pick) git push --force # Update the remote branch after rebaseReviewer ResponsibilitiesReviewers should check commit quality in addition to code quality.If a PR contains fragmented or trivial commits, the reviewer should request the contributor to clean them up before approving the PR.A PR with well-organized commit history makes it easier to understand the change, bisect bugs, and revert individual changes if needed.Relationship Between Platform Settings and Commit HygieneAspectWho is responsibleWhat happensDisable platform-level squashPlatform settingPrevents loss of author attributionMerge commit strategyPlatform settingPreserves full commit history and contribution statsCommit cleanup (squash trivial commits)Contributor (local)Ensures meaningful commit historyEnforce commit qualityReviewerEnsures PRs have clean, organized commitsSummary:The platform is configured to always use merge commits and disallow squash to preserve authorship. Contributors are responsible for keeping their own commit history clean and meaningful. Reviewers enforce this standard during code review.四、后端开发 (C)后端开发主要涉及在 C 层面实现 NPU 算子调用逻辑。本节以开发sinc函数为例。4.1 添加函数声明在对应的头文件中添加函数声明。sinc属于数学模块的特殊函数声明位于文件位置:include/asnumpy/math/other_special_functions.hpp/** * brief Compute the normalized sinc function element-wise on the input array. * * Uses NPU operator aclnnSinc to compute: * sinc(x) sin(pi * x) / (pi * x), with sinc(0) 1. * * param x Input array. * param dtype Optional output dtype. If not specified, uses input dtype. * return NPUArray Output array with sinc applied element-wise. * throws std::runtime_error If the ACL operator or memory allocation fails. */ NPUArray Sinc(const NPUArray x, std::optionalpy::dtype dtype std::nullopt);4.2 实现函数主体在对应的源文件中实现函数逻辑文件位置:src/math/other_special_functions.cppNPUArray Sinc(const NPUArray x, std::optionalpy::dtype dtype) { // 1. 确定输出数据类型 py::dtype py_dtype x.dtype; aclDataType in_dtype NPUArray::GetACLDataType(py_dtype); aclDataType out_dtype in_dtype; py::dtype out_py_dtype NPUArray::GetPyDtype(out_dtype); if (dtype ! std::nullopt) { out_py_dtype *dtype; out_dtype NPUArray::GetACLDataType(out_py_dtype); } // 2. 创建输出数组 NPUArray out(x.shape, out_py_dtype); // 3. 准备 NPU 算子执行所需资源 uint64_t workspaceSize 0; aclOpExecutor* executor nullptr; // 4. 获取 workspace 大小和执行器 auto error aclnnSincGetWorkspaceSize( x.tensorPtr, out.tensorPtr, workspaceSize, executor ); if (error ! ACL_SUCCESS) { std::string msg [other_special_functions.cpp](https://link.gitcode.com/i/2677876e473425c72f9fabbd55fb886c) aclnnSincGetWorkspaceSize error std::to_string(error); const char* detail aclGetRecentErrMsg(); if (detail std::strlen(detail) 0) msg - std::string(detail); throw std::runtime_error(msg); } // 5. 分配 workspace 内存如果需要 void* workspaceAddr nullptr; if (workspaceSize 0) { error aclrtMalloc(workspaceAddr, workspaceSize, ACL_MEM_MALLOC_HUGE_FIRST); if (error ! ACL_SUCCESS) { std::string msg [other_special_functions.cpp](https://link.gitcode.com/i/2677876e473425c72f9fabbd55fb886c) aclrtMalloc error std::to_string(error); throw std::runtime_error(msg); } } // 6. 执行算子 error aclnnSinc(workspaceAddr, workspaceSize, executor, nullptr); if (error ! ACL_SUCCESS) { if (workspaceAddr) aclrtFree(workspaceAddr); std::string msg [other_special_functions.cpp](https://link.gitcode.com/i/2677876e473425c72f9fabbd55fb886c) aclnnSinc error std::to_string(error); throw std::runtime_error(msg); } // 7. 同步设备确保计算完成 error aclrtSynchronizeDevice(); if (error ! ACL_SUCCESS) { if (workspaceAddr) aclrtFree(workspaceAddr); std::string msg [other_special_functions.cpp](https://link.gitcode.com/i/2677876e473425c72f9fabbd55fb886c) aclrtSynchronizeDevice error std::to_string(error); throw std::runtime_error(msg); } // 8. 释放 workspace 内存 if (workspaceAddr) { aclrtFree(workspaceAddr); } return out; }4.3 关键实现要点错误处理所有 ACL API 调用都需要检查返回值auto error aclnnSomeFunction(/* 参数 */); if (error ! ACL_SUCCESS) { std::string msg [filename](https://link.gitcode.com/i/8423c51eb3bd21d853f6f8de58b7af6c) aclnnSomeFunction error std::to_string(error); const char* detail aclGetRecentErrMsg(); if (detail std::strlen(detail) 0) { msg - std::string(detail); } throw std::runtime_error(msg); }NPU 算子执行流程典型的 NPU 算子执行流程包括GetWorkspaceSize: 获取所需 workspace 大小和执行器Malloc: 分配 workspace 内存如果需要Execute: 执行算子计算Synchronize: 同步设备等待计算完成Free: 释放 workspace 内存五、绑定层 (Pybind11)5.1 添加函数绑定在对应的绑定文件中添加函数绑定文件位置:python/bind_math.cppnamespace asnumpy { void bind_other_special_functions(py::module_ math); } void bind_math(py::module_ math) { math.doc() math module of asnumpy; bind_other_special_functions(math); } namespace asnumpy { void bind_other_special_functions(py::module_ math){ math.def(sinc, Sinc, py::arg(x), py::arg(dtype) py::none()); } }六、前端开发 (Python)前端开发主要涉及将 C 函数暴露到 Python 层并确保 API 与 NumPy 兼容。6.1 添加 Python 包装层在对应的 Python 模块中导入 C 函数并添加 Python 包装层文件位置:asnumpy/math.py首先从编译好的 C 扩展导入函数from .lib.asnumpy_core.math import ( sin as _ap_sin, cos as _ap_cos, sinc as _ap_sinc, # ... 其他函数 ) from .utils import ndarray, _convert_dtype然后为每个函数添加 Python 包装层def sinc(x: ndarray, dtype: Optional[np.dtype] None) - ndarray: return ndarray(_ap_sinc(x, _convert_dtype(dtype)))6.2 导出到主命名空间在主包的__init__.py中添加函数文件位置:asnumpy/__init__.pyfrom .math import ( sin, sinc, # ... 其他数学函数 ) __all__ [ # ... 其他导出 sin, sinc, ]七、测试编写asnumpy 使用 pytest 和自定义测试框架编写测试。7.1 测试文件组织测试文件按模块组织位于tests/asnumpy_tests/目录tests/ ├── conftest.py # pytest 配置 └── asnumpy_tests/ ├── math_tests/ │ └── test_miscellaneous.py # 测试用例 └── ...7.2 编写测试用例文件位置:tests/asnumpy_tests/math_tests/test_miscellaneous.pyimport numpy import pytest from asnumpy import testing def _create_array(xp, data, dtype): 辅助函数创建数组 np_arr numpy.array(data, dtypedtype) if xp is numpy: return np_arr return xp.ndarray.from_numpy(np_arr) testing.for_float_dtypes(no_float16True) testing.numpy_asnumpy_allclose(rtol1e-4, atol1e-5) def test_sinc_basic(xp, dtype): 基础随机测试 numpy.random.seed(42) np_a numpy.random.uniform(low-5.0, high5.0, size(10, 10)).astype(dtype) a _create_array(xp, np_a, dtype) return xp.sinc(a) testing.for_float_dtypes(no_float16True) testing.numpy_asnumpy_allclose(rtol1e-5, atol1e-8) def test_sinc_at_zero(xp, dtype): 测试 sinc 在 x0 处的行为 data [0.0, -0.0] a _create_array(xp, data, dtype) return xp.sinc(a)7.3 装饰器说明asnumpy 提供了丰富的测试装饰器用于参数化测试和结果比较。数据类型装饰器for_dtypes(dtypes)- 指定数据类型列表for_all_dtypes()- 所有数据类型默认排除不支持的 float16、uint32、uint64for_float_dtypes()- 浮点类型float32、float64for_int_dtypes()- 整数类型int8-64、uint8-32for_signed_dtypes()- 有符号整数int8-64for_unsigned_dtypes()- 无符号整数uint8-16for_complex_dtypes()- 复数类型complex64、complex128testing.for_float_dtypes(no_float16True) def test_func(xp, dtype): a xp.array([1.0, 2.0, 3.0], dtypedtype) return xp.sinc(a)内存顺序装饰器for_orders(orders)- 指定内存顺序列表for_cf_orders()- C 和 F 顺序testing.for_orders([C, F]) def test_func(xp, order): return xp.zeros((3, 3), orderorder)NumPy-Asnumpy 比较装饰器numpy_asnumpy_array_equal()- 比较数组是否完全相等numpy_asnumpy_allclose(rtol1e-7, atol0)- 比较浮点数组是否接近设置相对/绝对容差testing.numpy_asnumpy_allclose(rtol1e-5, atol1e-8) def test_func(xp, dtype): a xp.array([1.0, 2.0, 3.0], dtypedtype) return xp.sinc(a)pytest 集成装饰器parameterize(*params)- 参数化测试fixture- pytest fixtureskip- 跳过测试skipif(condition)- 条件跳过xfail- 预期失败testing.parameterize(n, [1, 2, 4]) def test_func(n): assert n 0八、编译与运行8.1 编译项目在开发模式下安装和编译项目pip install -e .8.2 运行测试运行所有测试pytest tests/运行特定模块测试# 数学模块测试 pytest tests/asnumpy_tests/math_tests/运行单个测试# 运行特定测试文件 pytest tests/asnumpy_tests/math_tests/test_miscellaneous.py # 运行特定测试函数 pytest tests/asnumpy_tests/math_tests/test_miscellaneous.py::test_sinc_basic附录常用命令速查# 安装项目 pip install -e . # 运行所有测试 pytest tests/ # 运行特定测试 pytest tests/asnumpy_tests/math_tests/test_miscellaneous.py # 详细输出 pytest -v # 显示打印输出 pytest -s # 清理并重新安装 pip uninstall asnumpy -y pip install -e .参考资源昇腾 CANN 文档NumPy 文档Pybind11 文档pytest 文档【免费下载链接】asnumpy哈尔滨工业大学计算学部苏统华、王甜甜老师团队联合华为CANN团队开发的华为昇腾NPU原生Numpy仓库项目地址: https://gitcode.com/cann/asnumpy创作声明:本文部分内容由AI辅助生成(AIGC),仅供参考