李宏毅机器学习笔记15

李宏毅机器学习笔记15强化学习基础与代码示例强化学习Reinforcement Learning, RL是机器学习的重要分支通过智能体与环境的交互学习最优策略。李宏毅教授的机器学习课程第15讲深入讲解了强化学习的基础概念、算法框架及实际应用。强化学习核心概念强化学习的核心要素包括智能体Agent决策主体通过动作与环境交互。环境Environment智能体交互的外部系统反馈状态和奖励。状态State环境在某一时刻的描述。动作Action智能体在状态下采取的行为。奖励Reward环境对智能体动作的即时反馈。目标是通过最大化累积奖励学习最优策略Policy即状态到动作的映射函数。马尔可夫决策过程MDP强化学习问题通常建模为马尔可夫决策过程包含状态空间 ( S )动作空间 ( A )转移概率 ( P(s|s,a) )奖励函数 ( R(s,a,s) )折扣因子 ( \gamma \in [0,1] )价值函数用于评估策略的优劣状态价值函数 ( V^\pi(s) \mathbb{E}\pi \left[ \sum{t0}^\infty \gamma^t r_t | s_0 s \right] )动作价值函数 ( Q^\pi(s,a) \mathbb{E}\pi \left[ \sum{t0}^\infty \gamma^t r_t | s_0 s, a_0 a \right] )策略梯度算法策略梯度方法直接优化策略参数通过梯度上升更新策略。其梯度公式为 [ \nabla_\theta J(\theta) \mathbb{E}\pi \left[ \nabla\theta \log \pi_\theta(a|s) Q^\pi(s,a) \right] ]以下是一个简单的策略梯度算法实现Python PyTorchimport torch import torch.optim as optim import numpy as np class PolicyNetwork(torch.nn.Module): def __init__(self, state_dim, action_dim): super().__init__() self.fc1 torch.nn.Linear(state_dim, 32) self.fc2 torch.nn.Linear(32, action_dim) def forward(self, x): x torch.relu(self.fc1(x)) return torch.softmax(self.fc2(x), dim-1) def train_policy_gradient(env, episodes1000, lr0.01, gamma0.99): state_dim env.observation_space.shape[0] action_dim env.action_space.n policy PolicyNetwork(state_dim, action_dim) optimizer optim.Adam(policy.parameters(), lrlr) for episode in range(episodes): state env.reset() log_probs [] rewards [] while True: state_tensor torch.FloatTensor(state) action_probs policy(state_tensor) action torch.multinomial(action_probs, 1).item() next_state, reward, done, _ env.step(action) log_prob torch.log(action_probs[action]) log_probs.append(log_prob) rewards.append(reward) state next_state if done: break # 计算折扣累积奖励 discounted_rewards [] cumulative_reward 0 for r in reversed(rewards): cumulative_reward r gamma * cumulative_reward discounted_rewards.insert(0, cumulative_reward) # 归一化奖励 discounted_rewards torch.FloatTensor(discounted_rewards) discounted_rewards (discounted_rewards - discounted_rewards.mean()) / (discounted_rewards.std() 1e-9) # 计算损失 policy_loss [] for log_prob, reward in zip(log_probs, discounted_rewards): policy_loss.append(-log_prob * reward) optimizer.zero_grad() torch.stack(policy_loss).sum().backward() optimizer.step()深度Q网络DQNDQN结合Q学习与深度学习通过经验回放和固定目标网络稳定训练。其更新公式为 [ Q(s,a) \leftarrow Q(s,a) \alpha \left( r \gamma \max_{a} Q(s,a) - Q(s,a) \right) ]DQN实现示例import torch import random from collections import deque class DQN(torch.nn.Module): def __init__(self, state_dim, action_dim): super().__init__() self.fc1 torch.nn.Linear(state_dim, 64) self.fc2 torch.nn.Linear(64, 64) self.fc3 torch.nn.Linear(64, action_dim) def forward(self, x): x torch.relu(self.fc1(x)) x torch.relu(self.fc2(x)) return self.fc3(x) def train_dqn(env, episodes1000, batch_size32, gamma0.99): state_dim env.observation_space.shape[0] action_dim env.action_space.n q_network DQN(state_dim, action_dim) target_network DQN(state_dim, action_dim) target_network.load_state_dict(q_network.state_dict()) optimizer optim.Adam(q_network.parameters(), lr0.001) replay_buffer deque(maxlen10000) for episode in range(episodes): state env.reset() episode_reward 0 while True: # ε-贪婪策略选择动作 if random.random() max(0.01, 0.1 - 0.01*episode/100): action env.action_space.sample() else: with torch.no_grad(): state_tensor torch.FloatTensor(state) action q_network(state_tensor).argmax().item() next_state, reward, done, _ env.step(action) replay_buffer.append((state, action, reward, next_state, done)) episode_reward reward state next_state # 经验回放 if len(replay_buffer) batch_size: batch random.sample(replay_buffer, batch_size) states, actions, rewards, next_states, dones zip(*batch) states torch.FloatTensor(np.array(states)) actions torch.LongTensor(np.array(actions)) rewards torch.FloatTensor(np.array(rewards)) next_states torch.FloatTensor(np.array(next_states)) dones torch.FloatTensor(np.array(dones)) # 计算目标Q值 with torch.no_grad(): target_q rewards gamma * (1 - dones) * target_network(next_states).max(1)[0] # 用心去关注细节让生活的每一天都充满希望行驶在未来的目标与理想交汇的路上。每一个夜晚都是思考的时刻勇于反省自我的过往才能在新的日出时看到更广阔的未来。在人海中追寻内心的宁静理解生命的真谛让爱与善良伴随光影绽放出人生的独特魅力。坚持不懈让生活的每一步都充满意义让心中不再有恐惧才能在挫折中不断崛起与成长。每一次微笑都是心灵的交流学会用爱去感染他人让生活因为关怀而变得温暖而充实。 https://blog.csdn.net/2601_95555772/article/details/159163137 https://blog.csdn.net/jdxm6zy0/article/details/159163139 https://blog.csdn.net/2601_95555789/article/details/159163138 https://blog.csdn.net/atnsn53w/article/details/159163141 https://blog.csdn.net/a21o3iaf/article/details/159163142 https://blog.csdn.net/l2cllx9t/article/details/159163145 https://blog.csdn.net/buff83ex/article/details/159163146 https://blog.csdn.net/ikaf73t7/article/details/159163150 https://blog.csdn.net/nyubjr0h/article/details/159163151 https://blog.csdn.net/2601_95555806/article/details/159163153 https://blog.csdn.net/fcbbory8/article/details/159163154 https://blog.csdn.net/2601_95555821/article/details/159163152 https://blog.csdn.net/m0fye0kn/article/details/159163156 https://blog.csdn.net/jb9wc2pf/article/details/159163158 https://blog.csdn.net/m7pvysil/article/details/159163157 https://blog.csdn.net/nyi627j9/article/details/159163160 https://blog.csdn.net/bqifbzk8/article/details/159163162 https://blog.csdn.net/hvawwu05/article/details/159163164 https://blog.csdn.net/pf0zqkrc/article/details/159163166 https://blog.csdn.net/yzkg9p56/article/details/159163163 https://blog.csdn.net/2601_95555803/article/details/159163167 https://blog.csdn.net/rcwyj5yw/article/details/159163161 https://blog.csdn.net/y7srviaq/article/details/159163168 https://blog.csdn.net/2601_95555802/article/details/159163169 https://blog.csdn.net/zow5qht6/article/details/159163172 https://blog.csdn.net/sqmb54i9/article/details/159163147 https://blog.csdn.net/gozsf05v/article/details/159163176 https://blog.csdn.net/2601_95555807/article/details/159163180 https://blog.csdn.net/jj6wv0k2/article/details/159163178 https://blog.csdn.net/2601_95555801/article/details/159163174 https://blog.csdn.net/2601_95555808/article/details/159163182 https://blog.csdn.net/opx84xf7/article/details/159163183 https://blog.csdn.net/xyv3sp3a/article/details/159163185 https://blog.csdn.net/oadwvztw/article/details/159163186 https://blog.csdn.net/u44jqouk/article/details/159163187 https://blog.csdn.net/2601_95555804/article/details/159163191 https://blog.csdn.net/2601_95555762/article/details/159163192 https://blog.csdn.net/2601_95555777/article/details/159163194 https://blog.csdn.net/ovjn8ow0/article/details/159163197 https://blog.csdn.net/2601_95555809/article/details/159163196 https://blog.csdn.net/2601_95555778/article/details/159163198 https://blog.csdn.net/2601_95555829/article/details/159163199 https://blog.csdn.net/p0fmpyyt/article/details/159163200 https://blog.csdn.net/l9w3aslk/article/details/159163205 https://blog.csdn.net/kd94k8xh/article/details/159163208 https://blog.csdn.net/2601_95555716/article/details/159163209 https://blog.csdn.net/w0trbykq/article/details/159163210 https://blog.csdn.net/qgh7l6uh/article/details/159163211 https://blog.csdn.net/2601_95555728/article/details/159163213 https://blog.csdn.net/2601_95555782/article/details/159163212 https://blog.csdn.net/d0ndc2s5/article/details/159163214 https://blog.csdn.net/ivwosn6b/article/details/159163216 https://blog.csdn.net/2601_95555724/article/details/159163217 https://blog.csdn.net/uhd3vfre/article/details/159163218 https://blog.csdn.net/xoyvxc45/article/details/159163221 https://blog.csdn.net/2601_95555722/article/details/159163219 https://blog.csdn.net/utoyxvui/article/details/159163222 https://blog.csdn.net/2601_95555720/article/details/159163225 https://blog.csdn.net/2601_95555729/article/details/159163226 https://blog.csdn.net/2601_95555776/article/details/159163230 https://blog.csdn.net/n0awdu1b/article/details/159163228 https://blog.csdn.net/z70813h9/article/details/159163232 https://blog.csdn.net/jl4j1olz/article/details/159163233 https://blog.csdn.net/r5v9xzma/article/details/159163234 https://blog.csdn.net/ontxyhyz/article/details/159163236 https://blog.csdn.net/sxccauxd/article/details/159163237 https://blog.csdn.net/2601_95555735/article/details/159163240 https://blog.csdn.net/u6co2uod/article/details/159163243 https://blog.csdn.net/hyn3zgnd/article/details/159163244 https://blog.csdn.net/2601_95555725/article/details/159163245 https://blog.csdn.net/pnhuf7ag/article/details/159163249 https://blog.csdn.net/2601_95555736/article/details/159163250 https://blog.csdn.net/cgq8o3eh/article/details/159163253 https://blog.csdn.net/axis11j2/article/details/159163248 https://blog.csdn.net/gnfhy4ba/article/details/159163258 https://blog.csdn.net/2601_95555743/article/details/159163262 https://blog.csdn.net/2601_95555717/article/details/159163263 https://blog.csdn.net/tnnl8nvv/article/details/159163265 https://blog.csdn.net/u1v9jp82/article/details/159163269 https://blog.csdn.net/ns87ewf7/article/details/159163270 https://blog.csdn.net/sh1iiqns/article/details/159163276 https://blog.csdn.net/buafzlon/article/details/159163277 https://blog.csdn.net/saz3c0gq/article/details/159163279 https://blog.csdn.net/jrj4swsm/article/details/159163281 https://blog.csdn.net/ovln1ss6/article/details/159163285 https://blog.csdn.net/qsqkgf5c/article/details/159163289 https://blog.csdn.net/w36pnbai/article/details/159163287 https://blog.csdn.net/ty2h5jim/article/details/159163291 https://blog.csdn.net/n5v70bxg/article/details/159163292 https://blog.csdn.net/aetke0y0/article/details/159163293 https://blog.csdn.net/qykzj6nd/article/details/159163299 https://blog.csdn.net/z87vpy0n/article/details/159163303 https://blog.csdn.net/l93kzmi0/article/details/159163304 https://blog.csdn.net/vyr4mylf/article/details/159163309 https://blog.csdn.net/2601_95555735/article/details/159163311 https://blog.csdn.net/qf1ubgcg/article/details/159163318 https://blog.csdn.net/aetke0y0/article/details/159163335 https://blog.csdn.net/qf1ubgcg/article/details/159163350 https://blog.csdn.net/n0awdu1b/article/details/159163366 https://blog.csdn.net/2601_95555743/article/details/159163369 https://blog.csdn.net/j0z5pk8q/article/details/159163371 https://blog.csdn.net/gyiy7ls9/article/details/159163374 https://blog.csdn.net/2601_95555782/article/details/159163375 https://blog.csdn.net/2601_95555760/article/details/159163377 https://blog.csdn.net/ns87ewf7/article/details/159163378 https://blog.csdn.net/zw4ovwhj/article/details/159163379 https://blog.csdn.net/jsgi4f2t/article/details/159163381 https://blog.csdn.net/2601_95555774/article/details/159163382 https://blog.csdn.net/cgq8o3eh/article/details/159163383 https://blog.csdn.net/uhd3vfre/article/details/159163384 https://blog.csdn.net/oka5um24/article/details/159163394 https://blog.csdn.net/2601_95555829/article/details/159163396 https://blog.csdn.net/rcwyj5yw/article/details/159163397 https://blog.csdn.net/hvawwu05/article/details/159163398 https://blog.csdn.net/gozsf05v/article/details/159163399 https://blog.csdn.net/q59elce4/article/details/159163401 https://blog.csdn.net/2601_95555793/article/details/159163403 https://blog.csdn.net/2601_95555762/article/details/159163405 https://blog.csdn.net/2601_95555806/article/details/159163413 https://blog.csdn.net/2601_95555809/article/details/159163422 https://blog.csdn.net/wncnm1q3/article/details/159163423 https://blog.csdn.net/m7pvysil/article/details/159163424 https://blog.csdn.net/2601_95555789/article/details/159163426 https://blog.csdn.net/2601_95555728/article/details/159163429 https://blog.csdn.net/jb9wc2pf/article/details/159163432 https://blog.csdn.net/itdwzttu/article/details/159163434 https://blog.csdn.net/mt7362r6/article/details/159163438 https://blog.csdn.net/na50ew37/article/details/159163439 https://blog.csdn.net/bqifbzk8/article/details/159163441 https://blog.csdn.net/u44jqouk/article/details/159163442 https://blog.csdn.net/atnsn53w/article/details/159163444 https://blog.csdn.net/e85pcf86/article/details/159163447 https://blog.csdn.net/wacyvuo2/article/details/159163449 https://blog.csdn.net/jj6wv0k2/article/details/159163448 https://blog.csdn.net/yzkg9p56/article/details/159163452 https://blog.csdn.net/oadwvztw/article/details/159163453 https://blog.csdn.net/2601_95555838/article/details/159163455 https://blog.csdn.net/sqmb54i9/article/details/159163457 https://blog.csdn.net/l2cllx9t/article/details/159163458 https://blog.csdn.net/ww8tgkcg/article/details/159163459 https://blog.csdn.net/2601_95555821/article/details/159163460 https://blog.csdn.net/ikaf73t7/article/details/159163462 https://blog.csdn.net/nyi627j9/article/details/159163464 https://blog.csdn.net/zow5qht6/article/details/159163463 https://blog.csdn.net/2601_95555803/article/details/159163468 https://blog.csdn.net/2601_95555794/article/details/159163466 https://blog.csdn.net/xyv3sp3a/article/details/159163469 https://blog.csdn.net/fcbbory8/article/details/159163471 https://blog.csdn.net/2601_95555801/article/details/159163472 https://blog.csdn.net/2601_95555799/article/details/159163474 https://blog.csdn.net/2601_95555802/article/details/159163475 https://blog.csdn.net/pf0zqkrc/article/details/159163483 https://blog.csdn.net/buff83ex/article/details/159163484 https://blog.csdn.net/2601_95555808/article/details/159163486 https://blog.csdn.net/2601_95555807/article/details/159163489 https://blog.csdn.net/hyn3zgnd/article/details/159163488 https://blog.csdn.net/2601_95555720/article/details/159163490 https://blog.csdn.net/2601_95555724/article/details/159163491 https://blog.csdn.net/z3om9ffv/article/details/159163492 https://blog.csdn.net/l9w3aslk/article/details/159163494 https://blog.csdn.net/in8i5nbq/article/details/159163493 https://blog.csdn.net/h7mntms2/article/details/159163495 https://blog.csdn.net/n5v70bxg/article/details/159163496 https://blog.csdn.net/a21o3iaf/article/details/159163498 https://blog.csdn.net/y7srviaq/article/details/159163499 https://blog.csdn.net/ty2h5jim/article/details/159163500 https://blog.csdn.net/opx84xf7/article/details/159163503 https://blog.csdn.net/2601_95555786/article/details/159163504 https://blog.csdn.net/2601_95555791/article/details/159163507 https://blog.csdn.net/qgh7l6uh/article/details/159163508 https://blog.csdn.net/2601_95555775/article/details/159163510 https://blog.csdn.net/2601_95555716/article/details/159163509 https://blog.csdn.net/vyr4mylf/article/details/159163512 https://blog.csdn.net/2601_95555717/article/details/159163513 https://blog.csdn.net/ontxyhyz/article/details/159163514 https://blog.csdn.net/xoyvxc45/article/details/159163515 https://blog.csdn.net/z87vpy0n/article/details/159163517 https://blog.csdn.net/olbtnjrv/article/details/159163518 https://blog.csdn.net/jdxm6zy0/article/details/159163520 https://blog.csdn.net/buafzlon/article/details/159163523 https://blog.csdn.net/qykzj6nd/article/details/159163529 https://blog.csdn.net/2601_95555729/article/details/159163527 https://blog.csdn.net/2601_95555772/article/details/159163528 https://blog.csdn.net/2601_95555730/article/details/159163531 https://blog.csdn.net/xzs16ffo/article/details/159163532 https://blog.csdn.net/utoyxvui/article/details/159163537 https://blog.csdn.net/u1v9jp82/article/details/159163540 https://blog.csdn.net/ovln1ss6/article/details/159163543 https://blog.csdn.net/nyubjr0h/article/details/159163544 https://blog.csdn.net/cuqlk1z9/article/details/159163551 https://blog.csdn.net/pnhuf7ag/article/details/159163555 https://blog.csdn.net/ovjn8ow0/article/details/159163556 https://blog.csdn.net/2601_95555804/article/details/159163558 https://blog.csdn.net/o4y65aeb/article/details/159163557 https://blog.csdn.net/sxccauxd/article/details/159163563 https://blog.csdn.net/d0ndc2s5/article/details/159163565 https://blog.csdn.net/saz3c0gq/article/details/159163566 https://blog.csdn.net/w36pnbai/article/details/159163569 https://blog.csdn.net/2601_95555778/article/details/159163570 https://blog.csdn.net/2601_95555736/article/details/159163571 https://blog.csdn.net/qz6em0lc/article/details/159163573 https://blog.csdn.net/fl5lhhdt/article/details/159163575 https://blog.csdn.net/quo9bz4q/article/details/159163577 https://blog.csdn.net/u6co2uod/article/details/159163581 https://blog.csdn.net/dyjyriem/article/details/159163591 https://blog.csdn.net/2601_95555753/article/details/159163592 https://blog.csdn.net/vd0sb5ti/article/details/159163572 https://blog.csdn.net/tnnl8nvv/article/details/159163599 https://blog.csdn.net/w0trbykq/article/details/159163598 https://blog.csdn.net/gnfhy4ba/article/details/159163603 https://blog.csdn.net/z70813h9/article/details/159163594 https://blog.csdn.net/ijcajcqi/article/details/159163608 https://blog.csdn.net/2601_95555725/article/details/159163611 https://blog.csdn.net/w5av9bjt/article/details/159163613 https://blog.csdn.net/jubdsyfy/article/details/159163616 https://blog.csdn.net/p0fmpyyt/article/details/159163617 https://blog.csdn.net/qtfe41ok/article/details/159163621 https://blog.csdn.net/2601_95555722/article/details/159163622 https://blog.csdn.net/2601_95555777/article/details/159163624 https://blog.csdn.net/ti1niriy/article/details/159163627 https://blog.csdn.net/h4ke7iek/article/details/159163666 https://blog.csdn.net/yg03abc1/article/details/159163729 https://blog.csdn.net/qbyvimkn/article/details/159163733 https://blog.csdn.net/mc9i58ki/article/details/159163734 https://blog.csdn.net/2601_95556019/article/details/159163735 https://blog.csdn.net/i5lxa8pu/article/details/159163736 https://blog.csdn.net/2601_95556043/article/details/159163738 https://blog.csdn.net/pxgl3kfg/article/details/159163740 https://blog.csdn.net/2601_95555942/article/details/159163749 https://blog.csdn.net/xppjt7as/article/details/159163751 https://blog.csdn.net/rfpdacwm/article/details/159163752 https://blog.csdn.net/ygk4mmy6/article/details/159163754 https://blog.csdn.net/qd4itlyb/article/details/159163760 https://blog.csdn.net/oay5vzmp/article/details/159163765 https://blog.csdn.net/czogt39k/article/details/159163766 https://blog.csdn.net/zok5zj0d/article/details/159163769 https://blog.csdn.net/hbnlxjdy/article/details/159163770 https://blog.csdn.net/f6uet9y5/article/details/159163771 https://blog.csdn.net/bey2bkj3/article/details/159163773 https://blog.csdn.net/lpi8nrwd/article/details/159163774 https://blog.csdn.net/gp46u7g2/article/details/159163775 https://blog.csdn.net/wnfcs5mj/article/details/159163776 https://blog.csdn.net/ffv8kec8/article/details/159163778 https://blog.csdn.net/pn81l1po/article/details/159163780 https://blog.csdn.net/shbnzc48/article/details/159163779 https://blog.csdn.net/2601_95555962/article/details/159163782 https://blog.csdn.net/o9jq60kh/article/details/159163783 https://blog.csdn.net/yd1urhdj/article/details/159163784 https://blog.csdn.net/nbmva245/article/details/159163785 https://blog.csdn.net/u81c9qbv/article/details/159163786 https://blog.csdn.net/u5qiq88o/article/details/159163789 https://blog.csdn.net/2601_95555928/article/details/159163788 https://blog.csdn.net/h2sox2v7/article/details/159163790 https://blog.csdn.net/oqek10iu/article/details/159163791 https://blog.csdn.net/pizv8f07/article/details/159163793 https://blog.csdn.net/g2t7fow1/article/details/159163794 https://blog.csdn.net/ybrkn8w3/article/details/159163795 https://blog.csdn.net/lo64zbvb/article/details/159163796 https://blog.csdn.net/su3d1itp/article/details/159163799 https://blog.csdn.net/2601_95555979/article/details/159163800 https://blog.csdn.net/bawqzhpn/article/details/159163797 https://blog.csdn.net/2601_95556004/article/details/159163801 https://blog.csdn.net/vzkbn8f7/article/details/159163802 https://blog.csdn.net/b3iy69sb/article/details/159163803 https://blog.csdn.net/t5d6888t/article/details/159163804 https://blog.csdn.net/2601_95555949/article/details/159163805 https://blog.csdn.net/2601_95555965/article/details/159163806 https://blog.csdn.net/vhlzvon4/article/details/159163808 https://blog.csdn.net/gjun70gh/article/details/159163810 https://blog.csdn.net/r8i6brc6/article/details/159163809 https://blog.csdn.net/x0rc3sht/article/details/159163811 https://blog.csdn.net/jn1j76ss/article/details/159163813 https://blog.csdn.net/wugezh9c/article/details/159163812 https://blog.csdn.net/2601_95555947/article/details/159163807 https://blog.csdn.net/j4lh0tck/article/details/159163814 https://blog.csdn.net/nw1ws93v/article/details/159163816 https://blog.csdn.net/peapcm0j/article/details/159163817 https://blog.csdn.net/svpl6xcd/article/details/159163819 https://blog.csdn.net/rm3ed31f/article/details/159163818 https://blog.csdn.net/2601_95556066/article/details/159163820 https://blog.csdn.net/gmz70tje/article/details/159163821 https://blog.csdn.net/w1bq5atw/article/details/159163823 https://blog.csdn.net/l7f7b4st/article/details/159163822 https://blog.csdn.net/2601_95556025/article/details/159163825 https://blog.csdn.net/escv5v68/article/details/159163824 https://blog.csdn.net/g6o4vu1u/article/details/159163827 https://blog.csdn.net/plytlxxo/article/details/159163826 https://blog.csdn.net/zv7g5kqa/article/details/159163829 https://blog.csdn.net/nj3fx3hu/article/details/159163828 https://blog.csdn.net/2601_95556022/article/details/159163830 https://blog.csdn.net/zg4gkfki/article/details/159163831 https://blog.csdn.net/u8xmuqxd/article/details/159163832 https://blog.csdn.net/r788burs/article/details/159163833 https://blog.csdn.net/2601_95556055/article/details/159163834 https://blog.csdn.net/hniinlhe/article/details/159163835 https://blog.csdn.net/ueasnj2s/article/details/159163837 https://blog.csdn.net/2601_95555996/article/details/159163838 https://blog.csdn.net/n9vxavbe/article/details/159163839 https://blog.csdn.net/2601_95555935/article/details/159163836 https://blog.csdn.net/2601_95555932/article/details/159163842 https://blog.csdn.net/xg3ro9y4/article/details/159163840 https://blog.csdn.net/w3jhnuc5/article/details/159163843 https://blog.csdn.net/kv3zvmpi/article/details/159163841 https://blog.csdn.net/hqzdjam6/article/details/159163845 https://blog.csdn.net/keoohmdp/article/details/159163844 https://blog.csdn.net/myge20tz/article/details/159163848 https://blog.csdn.net/2601_95555934/article/details/159163847 https://blog.csdn.net/vs97yety/article/details/159163849 https://blog.csdn.net/ebva43q1/article/details/159163850 https://blog.csdn.net/vc6d37o8/article/details/159163846 https://blog.csdn.net/sd2b2fun/article/details/159163851 https://blog.csdn.net/2601_95556021/article/details/159163852 https://blog.csdn.net/osixiwzf/article/details/159163853 https://blog.csdn.net/w4ylw7og/article/details/159163854 https://blog.csdn.net/m2fez4xf/article/details/159163855 https://blog.csdn.net/t2uzrs2c/article/details/159163856 https://blog.csdn.net/oqh22f1d/article/details/159163857 https://blog.csdn.net/zj9194sw/article/details/159163858 https://blog.csdn.net/dnhfmmqf/article/details/159163861 https://blog.csdn.net/hxip0rgm/article/details/159163860 https://blog.csdn.net/2601_95556045/article/details/159163862 https://blog.csdn.net/2601_95556023/article/details/159163863 https://blog.csdn.net/r5gppohp/article/details/159163859 https://blog.csdn.net/t972zthk/article/details/159163865 https://blog.csdn.net/lcl4onk3/article/details/159163867 https://blog.csdn.net/nr3vd6ja/article/details/159163864 https://blog.csdn.net/2601_95556002/article/details/159163866 https://blog.csdn.net/sdtrnlck/article/details/159163868 https://blog.csdn.net/p8eipe6b/article/details/159163869 https://blog.csdn.net/rg78agnh/article/details/159163870 https://blog.csdn.net/okye5c37/article/details/159163873 https://blog.csdn.net/2601_95556032/article/details/159163875 https://blog.csdn.net/2601_95556080/article/details/159163871 https://blog.csdn.net/wjal8e9e/article/details/159163872 https://blog.csdn.net/dyfp8k6l/article/details/159163874 https://blog.csdn.net/w53sn2xr/article/details/159163876 https://blog.csdn.net/c1pjjdql/article/details/159163878 https://blog.csdn.net/q8r1n1ir/article/details/159163879 https://blog.csdn.net/ay2acyy0/article/details/159163877 https://blog.csdn.net/hgab323a/article/details/159163880 https://blog.csdn.net/2601_95556065/article/details/159163881 https://blog.csdn.net/2601_95555955/article/details/159163883 https://blog.csdn.net/bh9bfm2x/article/details/159163884 https://blog.csdn.net/ttc96hlt/article/details/159163885 https://blog.csdn.net/sc4sqzem/article/details/159163886 https://blog.csdn.net/2601_95555939/article/details/159163882 https://blog.csdn.net/2601_95556063/article/details/159163889 https://blog.csdn.net/lzljw2r2/article/details/159163888 https://blog.csdn.net/2601_95555957/article/details/159163887 https://blog.csdn.net/zulz8jpt/article/details/159163890 https://blog.csdn.net/kyjs0xfq/article/details/159163892 https://blog.csdn.net/tp034klc/article/details/159163891 https://blog.csdn.net/urdzoj72/article/details/159163894 https://blog.csdn.net/v93i74kc/article/details/159163893 https://blog.csdn.net/u831yug7/article/details/159163895 https://blog.csdn.net/2601_95555953/article/details/159163897 https://blog.csdn.net/2601_95556019/article/details/159163898 https://blog.csdn.net/h56jkaem/article/details/159163896 https://blog.csdn.net/acpu37bt/article/details/159163900 https://blog.csdn.net/dudehmh4/article/details/159163901 https://blog.csdn.net/ypxy60e5/article/details/159163903 https://blog.csdn.net/b4mrj2yv/article/details/159163902 https://blog.csdn.net/p4908ujq/article/details/159163899 https://blog.csdn.net/wj9e5lzx/article/details/159163904 https://blog.csdn.net/2601_95556011/article/details/159163905 https://blog.csdn.net/i5pergpc/article/details/159163906 https://blog.csdn.net/luaspfhc/article/details/159163908 https://blog.csdn.net/2601_95556059/article/details/159163909 https://blog.csdn.net/ni9tdv68/article/details/159163910 https://blog.csdn.net/zsukwucd/article/details/159163911 https://blog.csdn.net/w6xe6hz1/article/details/159163912 https://blog.csdn.net/zrrpjo9b/article/details/159163914 https://blog.csdn.net/b2t8c3uz/article/details/159163915 https://blog.csdn.net/x62zzg9w/article/details/159163913 https://blog.csdn.net/2601_95555988/article/details/159163916 https://blog.csdn.net/2601_95556046/article/details/159163918 https://blog.csdn.net/oqcef44s/article/details/159163917 https://blog.csdn.net/lcwxgdbm/article/details/159163919 https://blog.csdn.net/2601_95556033/article/details/159163922 https://blog.csdn.net/2601_95556064/article/details/159163923 https://blog.csdn.net/fmgtc46o/article/details/159163925 https://blog.csdn.net/lcpyfvod/article/details/159163927 https://blog.csdn.net/ygk4mmy6/article/details/159163946 https://blog.csdn.net/rfpdacwm/article/details/159163956 https://blog.csdn.net/2601_95555962/article/details/159163990 https://blog.csdn.net/qd4itlyb/article/details/159163991 https://blog.csdn.net/n37ktlyj/article/details/159163995 https://blog.csdn.net/o9jq60kh/article/details/159164016 https://blog.csdn.net/oay5vzmp/article/details/159164025 https://blog.csdn.net/xp8du964/article/details/159164027 https://blog.csdn.net/czogt39k/article/details/159164026 https://blog.csdn.net/2601_95556060/article/details/159164035 https://blog.csdn.net/2601_95555979/article/details/159164037 https://blog.csdn.net/hbnlxjdy/article/details/159164042 https://blog.csdn.net/dnhfmmqf/article/details/159164045 https://blog.csdn.net/dnpo65b4/article/details/159164049 https://blog.csdn.net/ffv8kec8/article/details/159164050 https://blog.csdn.net/hxip0rgm/article/details/159164051 https://blog.csdn.net/nbmva245/article/details/159164052 https://blog.csdn.net/sd2b2fun/article/details/159164057