Lora假设原始权重矩阵是 W。 全量微调时我们直接学习一个新的权重更新量 ΔW于是模型里的权重会变成但大模型参数太多直接训练 ΔW 的代价很高。于是 LoRA 说 别学一个完整的大矩阵了我用两个小矩阵 A 和 B 相乘去近似这个更新量ΔW ≈ BA因为 A 和 B 的秩很低所以要训练的参数量会大幅下降。这就是 LoRA 能省显存、省训练成本的原因。LoRA 的论文和后续大量实践都证明它确实是非常有效的参数高效微调方法。❝W预训练参数冻结A ∈ ℝ^{r × d}降维矩阵B ∈ ℝ^{d × r}升维矩阵r秩rank远小于 d如 r4这样原本 768×768 的矩阵≈60万参数现在只需训练 768×4 4×768 ≈ 6000 参数 参数减少约 100×性能几乎不变。常规微调x.(WΔW) xW x.ΔWLora微调x·(WAB)xW xAB我们先简单的实现一下Loraimport torchimport torch.nn as nnclass LoRALinear(nn.Module): def __init__( self, base_layer: nn.Linear, rank: int 4, alpha: float 1.0, ): super().__init__() if not isinstance(base_layer, nn.Linear): raise TypeError(LoRALinear only supports nn.Linear) self.base_layer base_layer self.rank rank self.alpha alpha self.scaling alpha / rank in_features base_layer.in_features out_features base_layer.out_features # 冻结原始权重 self.base_layer.weight.requires_grad False if base_layer.bias is not None: base_layer.bias.requires_grad False # LoRA 参数低秩 self.lora_A nn.Linear(in_features, rank, biasFalse) self.lora_B nn.Linear(rank, out_features, biasFalse) # 初始化 nn.init.kaiming_uniform_(self.lora_A.weight, a5 ** 0.5) nn.init.zeros_(self.lora_B.weight) def forward(self, x): return self.base_layer(x) self.lora_B(self.lora_A(x)) * self.scaling❝W预训练参数冻结 A ∈ ℝ^{r × d}降维矩阵 B ∈ ℝ^{d × r}升维矩阵 r秩rank远小于 d如 r4现在将 LoRA 应用于Linear层我们发现结果相同因为我们还没有训练 LoRA 权重。torch.manual_seed(123)layer nn.Linear(10, 2,biasFalse)x torch.randn((1, 10))layer_lora_1 LoRALinear(layer,rank2, alpha4)print(layer_lora_1.lora_B(layer_lora_1.lora_A(x)))#tensor([[0., 0.]], grad_fnMmBackward0)print(Original output:, layer(x))#Original output: tensor([[ 0.4063, -0.5453]])print(LoRA output:, layer_lora_1(x))#LoRA output: tensor([[ 0.4063, -0.5453]], grad_fnAddBackward0)我们在一个由单层表示的小型神经网络层上尝试LoraLinear。 为了简单起见我们暂时只关注一个小型的 3 层多层感知器而不是 LLM如下图所示plaintextclass MultilayerPerceptron(nn.Module): definit(self, num_features, num_hidden_1, num_hidden_2, num_classes): super().init() self.layers nn.Sequential( nn.Linear(num_features, num_hidden_1), nn.ReLU(), nn.Linear(num_hidden_1, num_hidden_2), nn.ReLU(), nn.Linear(num_hidden_2, num_classes) ) def forward(self, x): x self.layers(x) return xmodel MultilayerPerceptron( num_features768, num_hidden_1128, num_hidden_2256, num_classes10)print(model)“”“MultilayerPerceptron( (layers): Sequential( (0): Linear(in_features768, out_features128, biasTrue) (1): ReLU() (2): Linear(in_features128, out_features256, biasTrue) (3): ReLU() (4): Linear(in_features256, out_features10, biasTrue) ))”“”利用该方法我们可以通过替换多层感知器模型中原始层的Linear添加 LoRA 层LoRALinear。 plaintext model.layers[0] LoRALinear(model.layers[0], rank4, alpha8)model.layers[2] LoRALinear(model.layers[2], rank4, alpha8)model.layers[4] LoRALinear(model.layers[4], rank4, alpha8)print(model)MultilayerPerceptron( (layers): Sequential( (0): LoRALinear( (base_layer): Linear(in_features768, out_features128, biasTrue) (lora_A): Linear(in_features768, out_features4, biasFalse) (lora_B): Linear(in_features4, out_features128, biasFalse) ) (1): ReLU() (2): LoRALinear( (base_layer): Linear(in_features128, out_features256, biasTrue) (lora_A): Linear(in_features128, out_features4, biasFalse) (lora_B): Linear(in_features4, out_features256, biasFalse) ) (3): ReLU() (4): LoRALinear( (base_layer): Linear(in_features256, out_features10, biasTrue) (lora_A): Linear(in_features256, out_features4, biasFalse) (lora_B): Linear(in_features4, out_features10, biasFalse) ) ))for name, param in model.named_parameters(): print(f{name}: {param.requires_grad})layers.0.base_layer.weight: Falselayers.0.base_layer.bias: Falselayers.0.lora_A.weight: Truelayers.0.lora_B.weight: Truelayers.2.base_layer.weight: Falselayers.2.base_layer.bias: Falselayers.2.lora_A.weight: Truelayers.2.lora_B.weight: Truelayers.4.base_layer.weight: Falselayers.4.base_layer.bias: Falselayers.4.lora_A.weight: Truelayers.4.lora_B.weight: True根据上述True和False值我们可以直观地确认目前只有 LoRA 层可训练True表示可训练False表示已冻结。但是 LoRA 有一个隐含问题 它主要是在对“更新矩阵”做低秩近似却没有显式区分“权重的长度变了多少”和“权重的方向转了多少”。而论文作者通过分析发现全量微调对这两部分都会进行调整而 LoRA 对“方向”的拟合更自然对“大小”的拟合相对不足。DoRA: Weight-Decomposed Low-Rank Adaptation在大模型微调里LoRA 很火因为它不用改全部参数只训练少量低秩矩阵就能让模型适配新任务。但问题也很明显LoRA 虽然省资源但有时候效果和全量微调之间还是会有一截差距。 DoRA 的出现就是想解决这个问题。 假设原始权重矩阵是 W。全量微调时我们直接学习一个新的权重更新量 ΔW。论文作者发现全量微调对向量的方向和大小都会进行调整而 LoRA 对“方向”的拟合更自然对“大小”的拟合相对不足。于是 DoRA 提出把权重分解成“模长magnitude”和“方向direction”两部分分开学。将向量分解为大小和方向分量其灵感来源于数学原理即任何向量都可以表示为其大小表示其长度的标量值与其方向表示其在空间中的方位的单位向量的乘积。所以DoRA 不再直接把权重更新写成一整个低秩增量而是先把预训练权重拆开一部分是 magnitude模长、大小一部分是 direction方向。 然后方向部分 仍然用 LoRA 那一套低秩方式去学习大小部分 单独用一个可学习参数来调这样一来模型既保留了低参数量微调的优点又能更细致地模拟全量微调的行为。LoRA和DoRA从头开始——一个多层感知器示例(使用LoRALinear和DoRALinear类)import timeimport numpy as npfrom torchvision import datasetsfrom torchvision import transformsfrom torch.utils.data import DataLoaderimport torch.nn.functional as Fimport torch.nn as nnimport torch# Settings and Dataset### SETTINGS# DeviceDEVICE torch.device(mpsif torch.backends.mps.is_available() elsecpu)BATCH_SIZE 256### MNIST DATASET# Note transforms.ToTensor() scales input images# to 0-1 rangedef load_data(batch_size256): transform transforms.ToTensor() train_data datasets.FashionMNIST(root./data, trainTrue, downloadTrue, transformtransform) test_data datasets.FashionMNIST(root./data, trainFalse, downloadTrue, transformtransform) return ( DataLoader(train_data, batch_sizebatch_size, shuffleTrue), DataLoader(test_data, batch_sizebatch_size) )train_loader,test_loader load_data(batch_sizeBATCH_SIZE)# Checking the datasetfor images, labels in train_loader: print(Image batch dimensions:, images.shape) print(Image label dimensions:, labels.shape) break### MODEL# Hyperparameterslearning_rate 0.005num_epochs 5# Architecturenum_features 784num_hidden_1 128num_hidden_2 256num_classes 10# Multilayer Perceptron Model (Without LoRA and DoRA)class MultilayerPerceptron(nn.Module): def __init__(self, num_features, num_hidden_1, num_hidden_2, num_classes): super().__init__() self.layers nn.Sequential( nn.Linear(num_features, num_hidden_1), nn.ReLU(), nn.Linear(num_hidden_1, num_hidden_2), nn.ReLU(), nn.Linear(num_hidden_2, num_classes) ) def forward(self, x): x self.layers(x) return xmodel_pretrained MultilayerPerceptron( num_featuresnum_features, num_hidden_1num_hidden_1, num_hidden_2num_hidden_2, num_classesnum_classes)model_pretrained.to(DEVICE)optimizer_pretrained torch.optim.Adam(model_pretrained.parameters(), lrlearning_rate)def compute_accuracy(model, data_loader, device): model.eval() correct_pred, num_examples 0, 0 with torch.no_grad(): for features, targets in data_loader: features features.view(-1, 28*28).to(device) targets targets.to(device) logits model(features) _, predicted_labels torch.max(logits, 1) num_examples targets.size(0) correct_pred (predicted_labels targets).sum() return correct_pred.float()/num_examples * 100def train(num_epochs, model, optimizer, train_loader, device): start_time time.time() for epoch in range(num_epochs): model.train() for batch_idx, (features, targets) in enumerate(train_loader): features features.view(-1, 28*28).to(device) targets targets.to(device) # FORWARD AND BACK PROP logits model(features) loss F.cross_entropy(logits, targets) optimizer.zero_grad() loss.backward() # UPDATE MODEL PARAMETERS optimizer.step() # LOGGING if not batch_idx % 400: loss_value loss.item() print(fEpoch {epoch 1}/{num_epochs} | Batch {batch_idx}/{len(train_loader)} | Loss {loss_value:.4f}) with torch.set_grad_enabled(False): print(Epoch: %03d/%03d training accuracy: %.2f%% % ( epoch1, num_epochs, compute_accuracy(model, train_loader, device))) print(fEpoch {epoch 1}/{num_epochs} | training accuracy {compute_accuracy(model, train_loader, device):.2f}) print(Time elapsed: %.2f min % ((time.time() - start_time)/60)) print(Total Training Time: %.2f min % ((time.time() - start_time)/60))train(num_epochs, model_pretrained, optimizer_pretrained, train_loader, DEVICE)print(fTest accuracy: {compute_accuracy(model_pretrained, test_loader, DEVICE):.2f}%)# 具有LoRA和DoRA的多层感知器,通过注入LoRA和DoRA层来修改模型class LoRALinear(nn.Module): def __init__( self, base_layer: nn.Linear, rank: int 4, alpha: float 1.0, ): super().__init__() if not isinstance(base_layer, nn.Linear): raise TypeError(LoRALinear only supports nn.Linear) self.base_layer base_layer self.rank rank self.alpha alpha self.scaling alpha / rank in_features base_layer.in_features out_features base_layer.out_features # 冻结原始权重 self.base_layer.weight.requires_grad False if base_layer.bias is not None: base_layer.bias.requires_grad False # LoRA 参数低秩 self.lora_A nn.Linear(in_features, rank, biasFalse) self.lora_B nn.Linear(rank, out_features, biasFalse) # 工业级初始化 nn.init.kaiming_uniform_(self.lora_A.weight, a5 ** 0.5) nn.init.zeros_(self.lora_B.weight) def forward(self, x): return self.base_layer(x) self.lora_B(self.lora_A(x)) * self.scalingclass DoRALinear(nn.Module): def __init__( self, base_layer: nn.Linear, rank: int 8, alpha: float 1.0, eps: float 1e-9, ): super().__init__() if not isinstance(base_layer, nn.Linear): raise TypeError(DoRALinear only supports nn.Linear) self.base_layer base_layer self.rank rank self.alpha alpha self.scaling alpha / rank self.eps eps in_features base_layer.in_features out_features base_layer.out_features # 冻结原始 Linear self.base_layer.weight.requires_grad False if base_layer.bias is not None: base_layer.bias.requires_grad False # LoRA 低秩分支 self.lora_A nn.Linear(in_features, rank, biasFalse) self.lora_B nn.Linear(rank, out_features, biasFalse) # DoRA 的 magnitude 参数每个输出通道一个 self.m nn.Parameter(torch.ones(out_features)) # 初始化工业标准 nn.init.kaiming_uniform_(self.lora_A.weight, a5 ** 0.5) nn.init.zeros_(self.lora_B.weight) def forward(self, x): # 原始线性输出 base_out self.base_layer(x) # LoRA 增量 lora_out self.lora_B(self.lora_A(x)) * self.scaling # 对 LoRA 输出做 L2 归一化按 token / batch 维 lora_norm lora_out.norm(p2, dim-1, keepdimTrue) lora_dir lora_out / (lora_norm self.eps) # DoRA用可学习的 magnitude 重新缩放方向 dora_out self.m * lora_dir return base_out dora_outmodel_lora MultilayerPerceptron( num_featuresnum_features, num_hidden_1num_hidden_1, num_hidden_2num_hidden_2, num_classesnum_classes)model_dora MultilayerPerceptron( num_featuresnum_features, num_hidden_1num_hidden_1, num_hidden_2num_hidden_2, num_classesnum_classes)def inject_adapter(model, adapter_cls, rank4, alpha8): for idx in [0, 2, 4]: model.layers[idx] adapter_cls(model.layers[idx], rankrank, alphaalpha) return modelDEVICE torch.device(mpsif torch.backends.mps.is_available() elsecpu)train_loader, test_loader load_data()torch.manual_seed(123)base_model MultilayerPerceptron(784, 128, 256, 10).to(DEVICE)optimizer torch.optim.Adam(base_model.parameters(), lr0.005)train(num_epochs, base_model, optimizer, train_loader, DEVICE)print(Base:, compute_accuracy(base_model, test_loader, DEVICE))model_lora inject_adapter( MultilayerPerceptron(784, 128, 256, 10), LoRALinear).to(DEVICE)optimizer_lora torch.optim.Adam(model_lora.parameters(), lr0.005)train(num_epochs, model_lora, optimizer_lora, train_loader, DEVICE)print(LoRA:, compute_accuracy(model_lora, test_loader, DEVICE))model_dora inject_adapter( MultilayerPerceptron(784, 128, 256, 10), DoRALinear).to(DEVICE)optimizer_dora torch.optim.Adam(model_dora.parameters(), lr0.005)train(num_epochs, model_dora, optimizer_dora, train_loader, DEVICE)print(DoRA:, compute_accuracy(model_dora, test_loader, DEVICE))在“小模型 小数据 从零训练”的场景下LoRA 往往会显著优于 DoRA。这个实验可能出现 LoRA 比 DoRA 好的结果。这是正确的而不是异常的。LoRA 在小模型 / 短训练 / 从零学习时更有优势DoRA 真正的舞台是“大模型 微调 稳定收敛”。总结在参数高效微调领域LoRA 之所以流行是因为它足够轻、足够实用而 DoRA 的价值在于它指出了 LoRA 的一个关键不足模型更新不只是方向问题还有尺度问题。 通过把权重分解为 magnitude 和 directionDoRA 在不明显增加推理成本的前提下让参数高效微调更接近全量微调的学习方式。简单说LoRA 解决的是“少量参数也能微调”DoRA 进一步解决的是“少量参数如何微调得更像全量训练”。AI行业迎来前所未有的爆发式增长从DeepSeek百万年薪招聘AI研究员到百度、阿里、腾讯等大厂疯狂布局AI Agent再到国家政策大力扶持数字经济和AI人才培养所有信号都在告诉我们AI的黄金十年真的来了在行业火爆之下AI人才争夺战也日趋白热化其就业前景一片蓝海我给大家准备了一份全套的《AI大模型零基础入门进阶学习资源包》包括AI大模型入门学习思维导图、精品AI大模型学习书籍手册、视频教程、实战学习等录播视频免费分享出来。有需要的小伙伴可以V扫描下方二维码免费领取人才缺口巨大人力资源社会保障部有关报告显示据测算当前****我国人工智能人才缺口超过500万****供求比例达1∶10。脉脉最新数据也显示AI新发岗位量较去年初暴增29倍超1000家AI企业释放7.2万岗位……单拿今年的秋招来说各互联网大厂释放出来的招聘信息中我们就能感受到AI浪潮比如百度90%的技术岗都与AI相关就业薪资超高在旺盛的市场需求下AI岗位不仅招聘量大薪资待遇更是“一骑绝尘”。企业为抢AI核心人才薪资给的非常慷慨过去一年懂AI的人才普遍涨薪40%脉脉高聘发布的《2025年度人才迁徙报告》显示在2025年1月-10月的高薪岗位Top20排行中AI相关岗位占了绝大多数并且平均薪资月薪都超过6w在去年的秋招中小红书给算法相关岗位的薪资为50k起字节开出228万元的超高年薪据《2025年秋季校园招聘白皮书》AI算法类平均年薪达36.9万遥遥领先其他行业总结来说当前人工智能岗位需求多薪资高前景好。在职场里选对赛道就能赢在起跑线。抓住AI风口轻松实现高薪就业但现实却是仍有很多同学不知道如何抓住AI机遇会遇到很多就业难题比如❌ 技术过时只会CRUD的开发者在AI浪潮中沦为“职场裸奔者”❌ 薪资停滞初级岗位内卷到白菜价传统开发3年经验薪资涨幅不足15%❌ 转型无门想学AI却找不到系统路径83%自学党中途放弃。他们的就业难题解决问题的关键在于不仅要选对赛道更要跟对老师我给大家准备了一份全套的《AI大模型零基础入门进阶学习资源包》包括AI大模型入门学习思维导图、精品AI大模型学习书籍手册、视频教程、实战学习等录播视频免费分享出来。有需要的小伙伴可以V扫描下方二维码免费领取
LoRA vs DoRA：揭秘大模型参数高效微调的终极奥义！
Lora假设原始权重矩阵是 W。 全量微调时我们直接学习一个新的权重更新量 ΔW于是模型里的权重会变成但大模型参数太多直接训练 ΔW 的代价很高。于是 LoRA 说 别学一个完整的大矩阵了我用两个小矩阵 A 和 B 相乘去近似这个更新量ΔW ≈ BA因为 A 和 B 的秩很低所以要训练的参数量会大幅下降。这就是 LoRA 能省显存、省训练成本的原因。LoRA 的论文和后续大量实践都证明它确实是非常有效的参数高效微调方法。❝W预训练参数冻结A ∈ ℝ^{r × d}降维矩阵B ∈ ℝ^{d × r}升维矩阵r秩rank远小于 d如 r4这样原本 768×768 的矩阵≈60万参数现在只需训练 768×4 4×768 ≈ 6000 参数 参数减少约 100×性能几乎不变。常规微调x.(WΔW) xW x.ΔWLora微调x·(WAB)xW xAB我们先简单的实现一下Loraimport torchimport torch.nn as nnclass LoRALinear(nn.Module): def __init__( self, base_layer: nn.Linear, rank: int 4, alpha: float 1.0, ): super().__init__() if not isinstance(base_layer, nn.Linear): raise TypeError(LoRALinear only supports nn.Linear) self.base_layer base_layer self.rank rank self.alpha alpha self.scaling alpha / rank in_features base_layer.in_features out_features base_layer.out_features # 冻结原始权重 self.base_layer.weight.requires_grad False if base_layer.bias is not None: base_layer.bias.requires_grad False # LoRA 参数低秩 self.lora_A nn.Linear(in_features, rank, biasFalse) self.lora_B nn.Linear(rank, out_features, biasFalse) # 初始化 nn.init.kaiming_uniform_(self.lora_A.weight, a5 ** 0.5) nn.init.zeros_(self.lora_B.weight) def forward(self, x): return self.base_layer(x) self.lora_B(self.lora_A(x)) * self.scaling❝W预训练参数冻结 A ∈ ℝ^{r × d}降维矩阵 B ∈ ℝ^{d × r}升维矩阵 r秩rank远小于 d如 r4现在将 LoRA 应用于Linear层我们发现结果相同因为我们还没有训练 LoRA 权重。torch.manual_seed(123)layer nn.Linear(10, 2,biasFalse)x torch.randn((1, 10))layer_lora_1 LoRALinear(layer,rank2, alpha4)print(layer_lora_1.lora_B(layer_lora_1.lora_A(x)))#tensor([[0., 0.]], grad_fnMmBackward0)print(Original output:, layer(x))#Original output: tensor([[ 0.4063, -0.5453]])print(LoRA output:, layer_lora_1(x))#LoRA output: tensor([[ 0.4063, -0.5453]], grad_fnAddBackward0)我们在一个由单层表示的小型神经网络层上尝试LoraLinear。 为了简单起见我们暂时只关注一个小型的 3 层多层感知器而不是 LLM如下图所示plaintextclass MultilayerPerceptron(nn.Module): definit(self, num_features, num_hidden_1, num_hidden_2, num_classes): super().init() self.layers nn.Sequential( nn.Linear(num_features, num_hidden_1), nn.ReLU(), nn.Linear(num_hidden_1, num_hidden_2), nn.ReLU(), nn.Linear(num_hidden_2, num_classes) ) def forward(self, x): x self.layers(x) return xmodel MultilayerPerceptron( num_features768, num_hidden_1128, num_hidden_2256, num_classes10)print(model)“”“MultilayerPerceptron( (layers): Sequential( (0): Linear(in_features768, out_features128, biasTrue) (1): ReLU() (2): Linear(in_features128, out_features256, biasTrue) (3): ReLU() (4): Linear(in_features256, out_features10, biasTrue) ))”“”利用该方法我们可以通过替换多层感知器模型中原始层的Linear添加 LoRA 层LoRALinear。 plaintext model.layers[0] LoRALinear(model.layers[0], rank4, alpha8)model.layers[2] LoRALinear(model.layers[2], rank4, alpha8)model.layers[4] LoRALinear(model.layers[4], rank4, alpha8)print(model)MultilayerPerceptron( (layers): Sequential( (0): LoRALinear( (base_layer): Linear(in_features768, out_features128, biasTrue) (lora_A): Linear(in_features768, out_features4, biasFalse) (lora_B): Linear(in_features4, out_features128, biasFalse) ) (1): ReLU() (2): LoRALinear( (base_layer): Linear(in_features128, out_features256, biasTrue) (lora_A): Linear(in_features128, out_features4, biasFalse) (lora_B): Linear(in_features4, out_features256, biasFalse) ) (3): ReLU() (4): LoRALinear( (base_layer): Linear(in_features256, out_features10, biasTrue) (lora_A): Linear(in_features256, out_features4, biasFalse) (lora_B): Linear(in_features4, out_features10, biasFalse) ) ))for name, param in model.named_parameters(): print(f{name}: {param.requires_grad})layers.0.base_layer.weight: Falselayers.0.base_layer.bias: Falselayers.0.lora_A.weight: Truelayers.0.lora_B.weight: Truelayers.2.base_layer.weight: Falselayers.2.base_layer.bias: Falselayers.2.lora_A.weight: Truelayers.2.lora_B.weight: Truelayers.4.base_layer.weight: Falselayers.4.base_layer.bias: Falselayers.4.lora_A.weight: Truelayers.4.lora_B.weight: True根据上述True和False值我们可以直观地确认目前只有 LoRA 层可训练True表示可训练False表示已冻结。但是 LoRA 有一个隐含问题 它主要是在对“更新矩阵”做低秩近似却没有显式区分“权重的长度变了多少”和“权重的方向转了多少”。而论文作者通过分析发现全量微调对这两部分都会进行调整而 LoRA 对“方向”的拟合更自然对“大小”的拟合相对不足。DoRA: Weight-Decomposed Low-Rank Adaptation在大模型微调里LoRA 很火因为它不用改全部参数只训练少量低秩矩阵就能让模型适配新任务。但问题也很明显LoRA 虽然省资源但有时候效果和全量微调之间还是会有一截差距。 DoRA 的出现就是想解决这个问题。 假设原始权重矩阵是 W。全量微调时我们直接学习一个新的权重更新量 ΔW。论文作者发现全量微调对向量的方向和大小都会进行调整而 LoRA 对“方向”的拟合更自然对“大小”的拟合相对不足。于是 DoRA 提出把权重分解成“模长magnitude”和“方向direction”两部分分开学。将向量分解为大小和方向分量其灵感来源于数学原理即任何向量都可以表示为其大小表示其长度的标量值与其方向表示其在空间中的方位的单位向量的乘积。所以DoRA 不再直接把权重更新写成一整个低秩增量而是先把预训练权重拆开一部分是 magnitude模长、大小一部分是 direction方向。 然后方向部分 仍然用 LoRA 那一套低秩方式去学习大小部分 单独用一个可学习参数来调这样一来模型既保留了低参数量微调的优点又能更细致地模拟全量微调的行为。LoRA和DoRA从头开始——一个多层感知器示例(使用LoRALinear和DoRALinear类)import timeimport numpy as npfrom torchvision import datasetsfrom torchvision import transformsfrom torch.utils.data import DataLoaderimport torch.nn.functional as Fimport torch.nn as nnimport torch# Settings and Dataset### SETTINGS# DeviceDEVICE torch.device(mpsif torch.backends.mps.is_available() elsecpu)BATCH_SIZE 256### MNIST DATASET# Note transforms.ToTensor() scales input images# to 0-1 rangedef load_data(batch_size256): transform transforms.ToTensor() train_data datasets.FashionMNIST(root./data, trainTrue, downloadTrue, transformtransform) test_data datasets.FashionMNIST(root./data, trainFalse, downloadTrue, transformtransform) return ( DataLoader(train_data, batch_sizebatch_size, shuffleTrue), DataLoader(test_data, batch_sizebatch_size) )train_loader,test_loader load_data(batch_sizeBATCH_SIZE)# Checking the datasetfor images, labels in train_loader: print(Image batch dimensions:, images.shape) print(Image label dimensions:, labels.shape) break### MODEL# Hyperparameterslearning_rate 0.005num_epochs 5# Architecturenum_features 784num_hidden_1 128num_hidden_2 256num_classes 10# Multilayer Perceptron Model (Without LoRA and DoRA)class MultilayerPerceptron(nn.Module): def __init__(self, num_features, num_hidden_1, num_hidden_2, num_classes): super().__init__() self.layers nn.Sequential( nn.Linear(num_features, num_hidden_1), nn.ReLU(), nn.Linear(num_hidden_1, num_hidden_2), nn.ReLU(), nn.Linear(num_hidden_2, num_classes) ) def forward(self, x): x self.layers(x) return xmodel_pretrained MultilayerPerceptron( num_featuresnum_features, num_hidden_1num_hidden_1, num_hidden_2num_hidden_2, num_classesnum_classes)model_pretrained.to(DEVICE)optimizer_pretrained torch.optim.Adam(model_pretrained.parameters(), lrlearning_rate)def compute_accuracy(model, data_loader, device): model.eval() correct_pred, num_examples 0, 0 with torch.no_grad(): for features, targets in data_loader: features features.view(-1, 28*28).to(device) targets targets.to(device) logits model(features) _, predicted_labels torch.max(logits, 1) num_examples targets.size(0) correct_pred (predicted_labels targets).sum() return correct_pred.float()/num_examples * 100def train(num_epochs, model, optimizer, train_loader, device): start_time time.time() for epoch in range(num_epochs): model.train() for batch_idx, (features, targets) in enumerate(train_loader): features features.view(-1, 28*28).to(device) targets targets.to(device) # FORWARD AND BACK PROP logits model(features) loss F.cross_entropy(logits, targets) optimizer.zero_grad() loss.backward() # UPDATE MODEL PARAMETERS optimizer.step() # LOGGING if not batch_idx % 400: loss_value loss.item() print(fEpoch {epoch 1}/{num_epochs} | Batch {batch_idx}/{len(train_loader)} | Loss {loss_value:.4f}) with torch.set_grad_enabled(False): print(Epoch: %03d/%03d training accuracy: %.2f%% % ( epoch1, num_epochs, compute_accuracy(model, train_loader, device))) print(fEpoch {epoch 1}/{num_epochs} | training accuracy {compute_accuracy(model, train_loader, device):.2f}) print(Time elapsed: %.2f min % ((time.time() - start_time)/60)) print(Total Training Time: %.2f min % ((time.time() - start_time)/60))train(num_epochs, model_pretrained, optimizer_pretrained, train_loader, DEVICE)print(fTest accuracy: {compute_accuracy(model_pretrained, test_loader, DEVICE):.2f}%)# 具有LoRA和DoRA的多层感知器,通过注入LoRA和DoRA层来修改模型class LoRALinear(nn.Module): def __init__( self, base_layer: nn.Linear, rank: int 4, alpha: float 1.0, ): super().__init__() if not isinstance(base_layer, nn.Linear): raise TypeError(LoRALinear only supports nn.Linear) self.base_layer base_layer self.rank rank self.alpha alpha self.scaling alpha / rank in_features base_layer.in_features out_features base_layer.out_features # 冻结原始权重 self.base_layer.weight.requires_grad False if base_layer.bias is not None: base_layer.bias.requires_grad False # LoRA 参数低秩 self.lora_A nn.Linear(in_features, rank, biasFalse) self.lora_B nn.Linear(rank, out_features, biasFalse) # 工业级初始化 nn.init.kaiming_uniform_(self.lora_A.weight, a5 ** 0.5) nn.init.zeros_(self.lora_B.weight) def forward(self, x): return self.base_layer(x) self.lora_B(self.lora_A(x)) * self.scalingclass DoRALinear(nn.Module): def __init__( self, base_layer: nn.Linear, rank: int 8, alpha: float 1.0, eps: float 1e-9, ): super().__init__() if not isinstance(base_layer, nn.Linear): raise TypeError(DoRALinear only supports nn.Linear) self.base_layer base_layer self.rank rank self.alpha alpha self.scaling alpha / rank self.eps eps in_features base_layer.in_features out_features base_layer.out_features # 冻结原始 Linear self.base_layer.weight.requires_grad False if base_layer.bias is not None: base_layer.bias.requires_grad False # LoRA 低秩分支 self.lora_A nn.Linear(in_features, rank, biasFalse) self.lora_B nn.Linear(rank, out_features, biasFalse) # DoRA 的 magnitude 参数每个输出通道一个 self.m nn.Parameter(torch.ones(out_features)) # 初始化工业标准 nn.init.kaiming_uniform_(self.lora_A.weight, a5 ** 0.5) nn.init.zeros_(self.lora_B.weight) def forward(self, x): # 原始线性输出 base_out self.base_layer(x) # LoRA 增量 lora_out self.lora_B(self.lora_A(x)) * self.scaling # 对 LoRA 输出做 L2 归一化按 token / batch 维 lora_norm lora_out.norm(p2, dim-1, keepdimTrue) lora_dir lora_out / (lora_norm self.eps) # DoRA用可学习的 magnitude 重新缩放方向 dora_out self.m * lora_dir return base_out dora_outmodel_lora MultilayerPerceptron( num_featuresnum_features, num_hidden_1num_hidden_1, num_hidden_2num_hidden_2, num_classesnum_classes)model_dora MultilayerPerceptron( num_featuresnum_features, num_hidden_1num_hidden_1, num_hidden_2num_hidden_2, num_classesnum_classes)def inject_adapter(model, adapter_cls, rank4, alpha8): for idx in [0, 2, 4]: model.layers[idx] adapter_cls(model.layers[idx], rankrank, alphaalpha) return modelDEVICE torch.device(mpsif torch.backends.mps.is_available() elsecpu)train_loader, test_loader load_data()torch.manual_seed(123)base_model MultilayerPerceptron(784, 128, 256, 10).to(DEVICE)optimizer torch.optim.Adam(base_model.parameters(), lr0.005)train(num_epochs, base_model, optimizer, train_loader, DEVICE)print(Base:, compute_accuracy(base_model, test_loader, DEVICE))model_lora inject_adapter( MultilayerPerceptron(784, 128, 256, 10), LoRALinear).to(DEVICE)optimizer_lora torch.optim.Adam(model_lora.parameters(), lr0.005)train(num_epochs, model_lora, optimizer_lora, train_loader, DEVICE)print(LoRA:, compute_accuracy(model_lora, test_loader, DEVICE))model_dora inject_adapter( MultilayerPerceptron(784, 128, 256, 10), DoRALinear).to(DEVICE)optimizer_dora torch.optim.Adam(model_dora.parameters(), lr0.005)train(num_epochs, model_dora, optimizer_dora, train_loader, DEVICE)print(DoRA:, compute_accuracy(model_dora, test_loader, DEVICE))在“小模型 小数据 从零训练”的场景下LoRA 往往会显著优于 DoRA。这个实验可能出现 LoRA 比 DoRA 好的结果。这是正确的而不是异常的。LoRA 在小模型 / 短训练 / 从零学习时更有优势DoRA 真正的舞台是“大模型 微调 稳定收敛”。总结在参数高效微调领域LoRA 之所以流行是因为它足够轻、足够实用而 DoRA 的价值在于它指出了 LoRA 的一个关键不足模型更新不只是方向问题还有尺度问题。 通过把权重分解为 magnitude 和 directionDoRA 在不明显增加推理成本的前提下让参数高效微调更接近全量微调的学习方式。简单说LoRA 解决的是“少量参数也能微调”DoRA 进一步解决的是“少量参数如何微调得更像全量训练”。AI行业迎来前所未有的爆发式增长从DeepSeek百万年薪招聘AI研究员到百度、阿里、腾讯等大厂疯狂布局AI Agent再到国家政策大力扶持数字经济和AI人才培养所有信号都在告诉我们AI的黄金十年真的来了在行业火爆之下AI人才争夺战也日趋白热化其就业前景一片蓝海我给大家准备了一份全套的《AI大模型零基础入门进阶学习资源包》包括AI大模型入门学习思维导图、精品AI大模型学习书籍手册、视频教程、实战学习等录播视频免费分享出来。有需要的小伙伴可以V扫描下方二维码免费领取人才缺口巨大人力资源社会保障部有关报告显示据测算当前****我国人工智能人才缺口超过500万****供求比例达1∶10。脉脉最新数据也显示AI新发岗位量较去年初暴增29倍超1000家AI企业释放7.2万岗位……单拿今年的秋招来说各互联网大厂释放出来的招聘信息中我们就能感受到AI浪潮比如百度90%的技术岗都与AI相关就业薪资超高在旺盛的市场需求下AI岗位不仅招聘量大薪资待遇更是“一骑绝尘”。企业为抢AI核心人才薪资给的非常慷慨过去一年懂AI的人才普遍涨薪40%脉脉高聘发布的《2025年度人才迁徙报告》显示在2025年1月-10月的高薪岗位Top20排行中AI相关岗位占了绝大多数并且平均薪资月薪都超过6w在去年的秋招中小红书给算法相关岗位的薪资为50k起字节开出228万元的超高年薪据《2025年秋季校园招聘白皮书》AI算法类平均年薪达36.9万遥遥领先其他行业总结来说当前人工智能岗位需求多薪资高前景好。在职场里选对赛道就能赢在起跑线。抓住AI风口轻松实现高薪就业但现实却是仍有很多同学不知道如何抓住AI机遇会遇到很多就业难题比如❌ 技术过时只会CRUD的开发者在AI浪潮中沦为“职场裸奔者”❌ 薪资停滞初级岗位内卷到白菜价传统开发3年经验薪资涨幅不足15%❌ 转型无门想学AI却找不到系统路径83%自学党中途放弃。他们的就业难题解决问题的关键在于不仅要选对赛道更要跟对老师我给大家准备了一份全套的《AI大模型零基础入门进阶学习资源包》包括AI大模型入门学习思维导图、精品AI大模型学习书籍手册、视频教程、实战学习等录播视频免费分享出来。有需要的小伙伴可以V扫描下方二维码免费领取
