OpenAI RLHF的理解

OpenAI RLHF的理解1. RLHF 的优化目标objectiveE(x,y)∼DπθRL[rθ(x,y)−βlog⁡πθ(y∣x)πref(y∣x)]γ Ex∼Dpretrain[log⁡πθRL(x)] \text{objective} \mathbb{E}_{(x,y) \sim D_{\pi_\theta^{RL}}} \left[ r_\theta(x, y) - \beta \log \frac{\pi_\theta(y \mid x)}{\pi_{\text{ref}}(y \mid x)} \right] \gamma \, \mathbb{E}_{x \sim D_{\text{pretrain}}} \left[ \log \pi_\theta^{RL}(x) \right]objectiveE(x,y)∼DπθRL​​​[rθ​(x,y)−βlogπref​(y∣x)πθ​(y∣x)​]γEx∼Dpretrain​​[logπθRL​(x)]2. 用 PPO-clip 来优化上述目标arg⁡max⁡θ E(x,y)∼Dπold[min⁡(πnew(y∣x)πold(y∣x)A, clip ⁣(πnew(y∣x)πold(y∣x), 1−ε, 1ε)A)] \arg\max_{\theta} \, \mathbb{E}_{(x,y) \sim D_{\pi_{old}}} \left[ \min \left( \frac{\pi_{\text{new}}(y \mid x)}{\pi_{\text{old}}(y \mid x)} A,\; \text{clip}\!\left( \frac{\pi_{\text{new}}(y \mid x)}{\pi_{\text{old}}(y \mid x)},\, 1-\varepsilon,\, 1\varepsilon \right) A \right) \right]argθmax​E(x,y)∼Dπold​​​[min(πold​(y∣x)πnew​(y∣x)​A,clip(πold​(y∣x)πnew​(y∣x)​,1−ε,1ε)A)]3. 为此需要计算如下变量需要估计以下三个量πnew(y∣x)\pi_{\text{new}}(y \mid x)πnew​(y∣x)当前策略在给定xxx下生成yyy的概率πold(y∣x)\pi_{\text{old}}(y \mid x)πold​(y∣x)旧策略在给定xxx下生成yyy的概率AAA优势函数Advantage变量说明在上述表达式中xxx是用户 promptyyy是生成的 tokenπ(y∣x)\pi(y \mid x)π(y∣x)是在策略π\piπ下promptxxx生成yyy的概率。优势函数定义AR−Value(x) A R - \text{Value}(x)AR−Value(x)其中Value\text{Value}Value是一个神经网络结构价值网络。4. 完整 RLHF 流程示例用一个具体例子描述完整的 RLHF 流程变量值xxx用户问“如何减肥”yyy回答“每天控制饮食并运动”πθ(y∣x)\pi_\theta(y \mid x)πθ​(y∣x)当前策略下已知xxx生成yyy的概率0.4πSFT(y∣x)\pi_{SFT}(y \mid x)πSFT​(y∣x)当SFT模型下已知xxx生成yyy的概率0.3Reward Model 打分rθ(x,y)0.8r_\theta(x, y) 0.8rθ​(x,y)0.8价值网络打分Value(x)0.2\text{Value}(x) 0.2Value(x)0.2计算 AdvantageKLlog⁡πθ(y∣x)πSFT(y∣x) KL \log \frac{\pi_{\theta}(y \mid x)}{\pi_{SFT}(y \mid x)}KLlogπSFT​(y∣x)πθ​(y∣x)​Arθ(x,y)−β∗KL−Value(x) A r_\theta(x, y) - \beta* KL -\text{Value}(x)Arθ​(x,y)−β∗KL−Value(x)至此πθ(y∣x)\pi_{\theta}(y \mid x)πθ​(y∣x)、πSFT(y∣x)\pi_{\text{SFT}}(y \mid x)πSFT​(y∣x)以及AAA全部就绪结合 Advantage 引入到 PPO-clip 中的所有变量均已确定。其中初始时令πnewπ训练时候(y∣x) \pi_{\text{new}} \pi_{\text{训练时候}}(y \mid x)πnew​π训练时候​(y∣x)πoldπrollout时模型(y∣x) \pi_{\text{old}} \pi_{\text{rollout时模型}}(y \mid x)πold​πrollout时模型​(y∣x)5. 用工程语言描述 RLHF 的完整流程涉及的模型模型作用输入输出Actor Model训练目标模型生成回复x→Actoryx \xrightarrow{\text{Actor}} yxActor​y输出P(y)P(y)P(y)的概率分布Reward Model打分模型(x,y)→Rewardr∼(0,1)(x, y) \xrightarrow{\text{Reward}} r \sim (0, 1)(x,y)Reward​r∼(0,1)Reference ModelSFT 基准模型用于计算 KL 散度x→Refyx \xrightarrow{\text{Ref}} yxRef​y输出P(y)P(y)P(y)概率Critic ModelValue 价值网络估计状态价值x→CriticV∼(0,1)x \xrightarrow{\text{Critic}} V \sim (0, 1)xCritic​V∼(0,1)Pretrain ModelOpenAI 的底层语言模型—步骤输入 prompt: “如何减肥”通过 Actor Model 生成 response“每天控制饮食并运动”得到πθ(y∣x)\pi_{\theta}(y \mid x)πθ​(y∣x)用 Reward Model 打分(x,y)→Reward Modelr(x, y) \xrightarrow{\text{Reward Model}} r(x,y)Reward Model​r计算 KL 散度用 Reference Model 算出rθ(x,y)r_\theta(x,y)rθ​(x,y),用sft模型得到πSFT(y∣x)\pi_{\text{SFT}}(y \mid x)πSFT​(y∣x)按 OpenAI 方式计算FinalRewardrθ(x,y)−β⋅KL ⁣(πθ∥πSFT)LLM \text{FinalReward} r_\theta(x,y) - \beta \cdot \text{KL}\!\left(\pi_\theta \| \pi_{\text{SFT}}\right) \mathcal{L}_{\text{LM}}FinalRewardrθ​(x,y)−β⋅KL(πθ​∥πSFT​)LLM​用 Critic Model 估算 Value 值VCritic(x)V \text{Critic}(x)VCritic(x)计算优势函数AAAAFinalReward−Critic Model 的值 A \text{FinalReward} - \text{Critic Model 的值}AFinalReward−Critic Model的值用 PPO-clip 更新策略loopπnew\pi_{\text{new}}πnew​训练时候一个epoch的更新后的模型用这个模型得到的这个 token 的输出概率πold\pi_{\text{old}}πold​Rollout时候的token输出概率参考[1] Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelley, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., Lowe, R. (2022). Training language models to follow instructions with human feedback.Advances in Neural Information Processing Systems, 35, 27730–27744. https://arxiv.org/abs/2203.02155