文章目录第4部kubernetes与devops整合kubernetes部署4初始化worker节点5检查初始化结果整合YAML文件准备harbor仓库对接YAML推送k8s服务器设置Jenkins使用ssh无密码登录访问k8s的master节点Jenkins中设置YAML部署第4部kubernetes与devops整合kubernetes部署快速安装kubernetes官网https://kuboard.cn/安装教程选择默认支持docker的版本1.191前置环境主机名解析[rootmaster ~]# echo 127.0.0.1 $(hostname) /etc/hosts[rootnode1 ~]# echo 127.0.0.1 $(hostname) /etc/hosts防火墙关闭禁用selinux[rootmaster ~]# systemctl disable firewalld.service --now[rootmaster ~]# setenforce 0[rootnode1 ~]# systemctl disable firewalld.service --now[rootnode1 ~]# setenforce 0配置固定IPmaster[rootmaster ~]# vim /etc/sysconfig/network-scripts/ifcfg-ens33TYPEEthernetPROXY_METHODnoneBROWSER_ONLYnoBOOTPROTOstaticDEFROUTEyesIPV4_FAILURE_FATALnoIPV6INITyesIPV6_AUTOCONFyesIPV6_DEFROUTEyesIPV6_FAILURE_FATALnoIPV6_ADDR_GEN_MODEstable-privacyNAMEens33UUID3abeb8f5-c5bd-4e3b-aefa-6410cdcc0d59DEVICEens33ONBOOTyesIPADDR192.168.108.32PREFIX24GATEWAY192.168.108.2DNS1192.168.108.2node1[rootnode1 ~]# vim /etc/sysconfig/network-scripts/ifcfg-ens33TYPEEthernetPROXY_METHODnoneBROWSER_ONLYnoBOOTPROTOstaticDEFROUTEyesIPV4_FAILURE_FATALnoIPV6INITyesIPV6_AUTOCONFyesIPV6_DEFROUTEyesIPV6_FAILURE_FATALnoIPV6_ADDR_GEN_MODEstable-privacyNAMEens33UUID3abeb8f5-c5bd-4e3b-aefa-6410cdcc0d59DEVICEens33ONBOOTyesIPADDR192.168.108.33PREFIX24GATEWAY192.168.108.2DNS1192.168.108.22安装docker及kubelet在所有节点上安装exportREGISTRY_MIRRORhttps://registry.cn-hangzhou.aliyuncs.comcurl-sSLhttps://kuboard.cn/install-script/v1.19.x/install_kubelet.sh|sh-s1.19.53初始化master节点在master节点上进行初始化修改export MASTER_IP192.168.108.32export APISERVER_NAMEabner.com其他内容保持不变进行复制粘贴# 只在 master 节点执行# 替换 x.x.x.x 为 master 节点实际 IP请使用内网 IP# export 命令只在当前 shell 会话中有效开启新的 shell 窗口后如果要继续安装过程请重新执行此处的 export 命令[rootmaster ~]# export MASTER_IP192.168.108.32# 替换 apiserver.demo 为 您想要的 dnsName[rootmaster ~]# export APISERVER_NAMEabner.com# Kubernetes 容器组所在的网段该网段安装完成后由 kubernetes 创建事先并不存在于您的物理网络中[rootmaster ~]# export POD_SUBNET10.100.0.1/16[rootmaster ~]# echo ${MASTER_IP} ${APISERVER_NAME} /etc/hosts[rootmaster ~]# curl -sSL https://kuboard.cn/install-script/v1.19.x/init_master.sh | sh -s 1.19.5初始化完成显示检查初始化结果watchkubectl get pod-nkube-system-owide需要等待全部为running需要开启加速等待时间十分钟左右若在 停留过长下载失败# 通过日志查看错误[rootmaster ~]# journalctl -u kubelet --since 2 minutes ago --no-pager | grep -E Error|Failed|Fatal|cgroup|driverFeb2516:28:31 master kubelet[9500]: E022516:28:31.1333169500reflector.go:127]k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed towatch*v1.Pod: failed to list *v1.Pod: Gethttps://abner.com:6443/api/v1/pods?fieldSelectorspec.nodeName%3Dmasterlimit500resourceVersion0:dial tcp216.40.34.37:6443: connect: connection refused Feb2516:28:31 master kubelet[9500]: E022516:28:31.9334559500reflector.go:127]k8s.io/client-go/informers/factory.go:134: Failed towatch*v1.Service: failed to list *v1.Service: Gethttps://abner.com:6443/api/v1/services?limit500resourceVersion0:dial tcp216.40.34.37:6443: connect: connection refused# DNS解析错误解决方法# 获取本机内网 IPLOCAL_IP192.168.108.32# 备份 hostscp/etc/hosts /etc/hosts.bak# 删除原有的错误解析 (如果有)sed-i/abner.com/d/etc/hostssed-i/^.*master$/d/etc/hosts# 添加正确的解析echo$LOCAL_IPmaster abner.com kubernetes kubernetes.default/etc/hosts# 验证cat/etc/hostsping-c1abner.comping-c1master清理残留环境# 1. 重置 kubeadmkubeadm reset-f# 2. 清理残留的 etcd 数据和 pki 证书 (因为证书里绑定了错误的 DNS/IP 组合建议重新生成)rm-rf/var/lib/etcd/*rm-rf/etc/kubernetes/pki/*# 注意保留 /etc/kubernetes/pki/etcd 如果不想重签 etcd 证书也可以但为了干净起见全删让脚本重生成最稳妥。# 这里我们只删 pki 目录下的内容保留目录结构rm-rf/etc/kubernetes/manifests/*rm-rf/var/lib/kubelet/*# 3. 重启 kubeletsystemctl restart kubelet查看node状态[rootmaster ~]# kubectl get nodes -o wideNAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME master Ready master 117m v1.19.5192.168.108.32noneCentOS Linux7(Core)3.10.0-1160.el7.x86_64 docker://19.3.114初始化worker节点获取join命令参数在master节点执行生成的token有效时间为2小时[rootmaster ~]# kubeadm token create --print-join-commandW022518:32:19.939401119982configset.go:348]WARNING: kubeadm cannot validate component configsforAPIgroups[kubelet.config.k8s.io kubeproxy.config.k8s.io]kubeadmjoinabner.com:6443--token19zdch.n9wx51hcnxasftfu --discovery-token-ca-cert-hash sha256:a466de0689ac189193277c99f61e8ac36b243c615324937276f430d460b59671针对所有worker节点执行# 只在 worker 节点执行# 替换 x.x.x.x 为 master 节点的内网 IP[rootnode1 ~]# export MASTER_IP192.168.108.32# 替换 apiserver.demo 为初始化 master 节点时所使用的 APISERVER_NAME[rootnode1 ~]# export APISERVER_NAMEabner.com[rootnode1 ~]# echo ${MASTER_IP} ${APISERVER_NAME} /etc/hosts[rootnode1 ~]# kubeadm join abner.com:6443 --token 19zdch.n9wx51hcnxasftfu --discovery-token-ca-cert-hash sha256:a466de0689ac189193277c99f61e8ac36b243c615324937276f430d460b59671初始化完成信息5检查初始化结果需要等待几分钟worker节点为Ready状态[rootmaster ~]# kubectl get nodeNAME STATUS ROLES AGE VERSION master Ready master 134m v1.19.5 node1 Readynone13m v1.19.5整合YAML文件准备因为yaml资源文件可以在k8s上创建应用pod所以需要提前创建yaml拉取到k8s服务器上在gitlab上新建YAML文件输入YAML文件内容apiVersion: apps/v1 kind: Deployment metadata: name: pipeline labels: app: pipeline spec: replicas:2selector: matchLabels: app: pipeline template: metadata: labels: app: pipeline spec: containers: - name: pipeline image:192.168.108.31:80/repo/pipeline:v4.0 imagePullPolicy: Always# 保持最新版本ports: - containerPort:8080--- apiVersion: v1 kind: Service metadata: labels: app: pipeline name: pipeline spec: selector: app: pipeline ports: - port:8081targetPort:8080type: NodePortharbor仓库对接在所有k8smasternode节点添加harbor仓库地址配置[rootmaster ~]# vim /etc/docker/daemon.json{insecure-registries:[192.168.108.31:80],registry-mirrors:[https://registry.cn-hangzhou.aliyuncs.com],exec-opts:[native.cgroupdriversystemd],log-driver:json-file,log-opts:{max-size:100m},storage-driver:overlay2,storage-opts:[overlay2.override_kernel_checktrue]}重启docker服务[rootmaster ~]# systemctl restart docker测试k8smasternode节点harbor登录[rootmaster ~]# docker login 192.168.108.31:80 -u admin -p harbor123WARNING!Using--passwordvia the CLI is insecure. Use --password-stdin. WARNING!Your password will be stored unencryptedin/root/.docker/config.json. Configure a credential helper to remove this warning. See https://docs.docker.com/engine/reference/commandline/login/#credentials-storeLogin SucceededYAML推送k8s服务器设置在master节点配置推送目标目录[rootmaster ~]# mkdir /usr/local/k8s[rootmaster ~]# chmod 777 /usr/local/k8s系统管理-系统配置点击测试成功后应用并保存在流水线中修改Publish Over SSH通知目标服务器步骤的Jenkinsfile的内容到gitlab服务器上修改Jenkinsfile内容保存配置并添加新标签tag否则识别不到新添加的yml文件先构建测试yaml文件推送查看构建日志构建失败原因Jenkins执行docker命令权限不足解决Jenkins节点执行以下命令cd/var/runchownroot:root docker.sockchmodorw docker.sock到master节点上查看已成功传递[rootmaster ~]# ls /usr/local/k8spipeline.yamlJenkins使用ssh无密码登录访问k8s的master节点因为希望使用ssh 用户名k8s地址 kubectl apply -f /usr/local/k8s/pipeline.yml创建资源会进行ssh 免交互把Jenkins中的公钥内容传递给k8s的master实现免密[rootjenkins ~]# docker exec -it jenkins bashjenkins51c548a67cdf:/$cd~ jenkins51c548a67cdf:~$ ssh-keygen-trsa Generating public/private rsa key pair. Enterfileinwhichto save the key(/var/jenkins_home/.ssh/id_rsa): Created directory/var/jenkins_home/.ssh.Enter passphrase(emptyforno passphrase): Enter same passphrase again: Your identification has been savedin/var/jenkins_home/.ssh/id_rsa Your public key has been savedin/var/jenkins_home/.ssh/id_rsa.pub The key fingerprint is: SHA256:0LH7tTXQ4j3nnD0wPBalGlIS4yKD67zJl91L/TkG4Y jenkins51c548a67cdf The keys randomart image is: ---[RSA3072]----|....||...o .o||.o..oo .oo.||.o......||.S. oB.||...o.Bo*o||...E.* o||..o. o o.||.|----[SHA256]-----jenkins51c548a67cdf:~$cd.ssh/ jenkins51c548a67cdf:~/.ssh$lsid_rsa id_rsa.pub复制公钥内容传递给masterjenkins51c548a67cdf:~/.ssh$catid_rsa.pub ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDuN33/QHO/MKpGnLT8n39qSGEHU7OPAaTZl9dQMWCFezM8WtLUJN7fUzOVixy1W2NKPSEckAMT1UTm8kLJM7yOAT/50bfVQvhM1LLjwPbIqrt4CeT/Hi0WhvBu9TzBZXuO88c/EOOO5mhBLQYOzHlGmu1gGlNORBYJRhyXQWUIPkMbH6jCYP1MCKLF85x5F/6bCsF65HhkLmf/i5XYDmYJeHMpeiHPhSfSgDtJFHd7Fq3XTfy4r9vrMthbpcpED3fpcqErXLY/m1V3QbJKC7szwsWjLzLeyIO2EYro7hIG9dEewlNeho57doL99NJkIL0jo27cqdvpBPwvP4niBKOratl/Wo6udeZBhBAi93nx/B5HE2ZjQ2iYYH6TF/USsWqs2YtFEv3yNaRmWGPg/VGojk99MpN2QgSCtw3Cumz4L3m/plPSmKErW39Xmbz3yFbUbNccgpnEhGvnZtQvk7m/unOLSFLI5MQ9P1k7iQ3fWRvNgXOTl8Yocjenkins51c548a67cdf到master节点上[rootmaster ~]# mkdir .ssh[rootmaster ~]# cd .ssh/[rootmaster .ssh]# vim authorized_keysssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDuN33/QHO/MKpGnLT8n39qSGEHU7OPAaTZl9dQMWCFezM8WtLUJN7fUzOVixy1W2NKPSEckAMT1UTm8kLJM7yOAT/50bfVQvhM1LLjwPbIqrt4CeT/Hi0WhvBu9TzBZXuO88c/EOOO5mhBLQYOzHlGmu1gGlNORBYJRhyXQWUIPkMbH6jCYP1MCKLF85x5F/6bCsF65HhkLmf/i5XYDmYJeHMpeiHPhSfSgDtJFHd7Fq3XTfy4r9vrMthbpcpED3fpcqErXLY/m1V3QbJKC7szwsWjLzLeyIO2EYro7hIG9dEewlNeho57doL99NJkIL0jo27cqdvpBPwvP4niBKOratl/Wo6udeZBhBAi93nx/B5HE2ZjQ2iYYH6TF/USsWqs2YtFEv3yNaRmWGPg/VGojk99MpN2QgSCtw3Cumz4L3m/plPSmKErW39Xmbz3yFbUbNccgpnEhGvnZtQvk7m/unOLSFLI5MQ9P1k7iQ3fWRvNgXOTl8Yocjenkins51c548a67cdfmaster重启sshd服务[rootmaster .ssh]# systemctl restart sshd在Jenkins容器中测试免密执行jenkins51c548a67cdf:~/.ssh$sshroot192.168.108.32ls/ bin boot dev etc home lib lib64 media mnt opt proc root run sbin srv sys tmp usr varJenkins中设置YAML部署在流水线语法中输入注意执行的是绝对路径sshroot192.168.108.32 kubectl apply-f/usr/local/k8s/pipeline.yaml在gitlab中增加步骤增加步骤内容stage(远程执行kubectl命令){steps{shssh root192.168.108.32 kubectl apply -f /usr/local/k8s/pipeline.yaml}}因为拉取的项目标签是4.0所以pipeline.yml拉取镜像的标签也要同步变更首页内容变更路径src/main/java/com/guo/demo/controller/Test/Controller.javagitlab中把原来的v4.0标签删除重新创建命名依然写v4.0立即构建在master上查看pod资源[rootmaster .ssh]# kubectl get pod -o wideNAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES pipeline-6b4965b559-657dm1/1 Running0112s10.100.166.129 node1nonenonepipeline-6b4965b559-mbj7p1/1 Running0112s10.100.166.130 node1nonenone[rootmaster .ssh]# kubectl get svcNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)AGE kubernetes ClusterIP10.96.0.1none443/TCP 18h pipeline NodePort10.96.53.129none8081:31463/TCP 2m9s在node1上查看下载的pipeline镜像[rootnode1 ~]# docker imagesREPOSITORY TAG IMAGE ID CREATED SIZE192.168.108.31:80/repo/pipeline v4.0 1332856cac9d2minutes ago 543MB[rootnode1 ~]# docker ps -a | grep pipeline29dc0d1c1fc5192.168.108.31:80/repo/pipeline/bin/sh -c java -j…2minutes ago Up2minutes k8s_pipeline_pipeline-6b4965b559-mbj7p_default_90e329ee-1d45-4c20-90e1-9a5ec26d30fa_0 81dd1211dbe2192.168.108.31:80/repo/pipeline/bin/sh -c java -j…2minutes ago Up2minutes k8s_pipeline_pipeline-6b4965b559-657dm_default_cef13886-b974-426d-ba87-bab6e11c80f8_0 bee3825f8892 registry.aliyuncs.com/k8sxio/pause:3.2/pause2minutes ago Up2minutes k8s_POD_pipeline-6b4965b559-mbj7p_default_90e329ee-1d45-4c20-90e1-9a5ec26d30fa_0 6d6bfdab414c registry.aliyuncs.com/k8sxio/pause:3.2/pause2minutes ago Up2minutes k8s_POD_pipeline-6b4965b559-657dm_default_cef13886-b974-426d-ba87-bab6e11c80f8_0打开node地址查看网页http://192.168.108.33:31463/test查看钉钉通知
60.devops-kubernetes
文章目录第4部kubernetes与devops整合kubernetes部署4初始化worker节点5检查初始化结果整合YAML文件准备harbor仓库对接YAML推送k8s服务器设置Jenkins使用ssh无密码登录访问k8s的master节点Jenkins中设置YAML部署第4部kubernetes与devops整合kubernetes部署快速安装kubernetes官网https://kuboard.cn/安装教程选择默认支持docker的版本1.191前置环境主机名解析[rootmaster ~]# echo 127.0.0.1 $(hostname) /etc/hosts[rootnode1 ~]# echo 127.0.0.1 $(hostname) /etc/hosts防火墙关闭禁用selinux[rootmaster ~]# systemctl disable firewalld.service --now[rootmaster ~]# setenforce 0[rootnode1 ~]# systemctl disable firewalld.service --now[rootnode1 ~]# setenforce 0配置固定IPmaster[rootmaster ~]# vim /etc/sysconfig/network-scripts/ifcfg-ens33TYPEEthernetPROXY_METHODnoneBROWSER_ONLYnoBOOTPROTOstaticDEFROUTEyesIPV4_FAILURE_FATALnoIPV6INITyesIPV6_AUTOCONFyesIPV6_DEFROUTEyesIPV6_FAILURE_FATALnoIPV6_ADDR_GEN_MODEstable-privacyNAMEens33UUID3abeb8f5-c5bd-4e3b-aefa-6410cdcc0d59DEVICEens33ONBOOTyesIPADDR192.168.108.32PREFIX24GATEWAY192.168.108.2DNS1192.168.108.2node1[rootnode1 ~]# vim /etc/sysconfig/network-scripts/ifcfg-ens33TYPEEthernetPROXY_METHODnoneBROWSER_ONLYnoBOOTPROTOstaticDEFROUTEyesIPV4_FAILURE_FATALnoIPV6INITyesIPV6_AUTOCONFyesIPV6_DEFROUTEyesIPV6_FAILURE_FATALnoIPV6_ADDR_GEN_MODEstable-privacyNAMEens33UUID3abeb8f5-c5bd-4e3b-aefa-6410cdcc0d59DEVICEens33ONBOOTyesIPADDR192.168.108.33PREFIX24GATEWAY192.168.108.2DNS1192.168.108.22安装docker及kubelet在所有节点上安装exportREGISTRY_MIRRORhttps://registry.cn-hangzhou.aliyuncs.comcurl-sSLhttps://kuboard.cn/install-script/v1.19.x/install_kubelet.sh|sh-s1.19.53初始化master节点在master节点上进行初始化修改export MASTER_IP192.168.108.32export APISERVER_NAMEabner.com其他内容保持不变进行复制粘贴# 只在 master 节点执行# 替换 x.x.x.x 为 master 节点实际 IP请使用内网 IP# export 命令只在当前 shell 会话中有效开启新的 shell 窗口后如果要继续安装过程请重新执行此处的 export 命令[rootmaster ~]# export MASTER_IP192.168.108.32# 替换 apiserver.demo 为 您想要的 dnsName[rootmaster ~]# export APISERVER_NAMEabner.com# Kubernetes 容器组所在的网段该网段安装完成后由 kubernetes 创建事先并不存在于您的物理网络中[rootmaster ~]# export POD_SUBNET10.100.0.1/16[rootmaster ~]# echo ${MASTER_IP} ${APISERVER_NAME} /etc/hosts[rootmaster ~]# curl -sSL https://kuboard.cn/install-script/v1.19.x/init_master.sh | sh -s 1.19.5初始化完成显示检查初始化结果watchkubectl get pod-nkube-system-owide需要等待全部为running需要开启加速等待时间十分钟左右若在 停留过长下载失败# 通过日志查看错误[rootmaster ~]# journalctl -u kubelet --since 2 minutes ago --no-pager | grep -E Error|Failed|Fatal|cgroup|driverFeb2516:28:31 master kubelet[9500]: E022516:28:31.1333169500reflector.go:127]k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed towatch*v1.Pod: failed to list *v1.Pod: Gethttps://abner.com:6443/api/v1/pods?fieldSelectorspec.nodeName%3Dmasterlimit500resourceVersion0:dial tcp216.40.34.37:6443: connect: connection refused Feb2516:28:31 master kubelet[9500]: E022516:28:31.9334559500reflector.go:127]k8s.io/client-go/informers/factory.go:134: Failed towatch*v1.Service: failed to list *v1.Service: Gethttps://abner.com:6443/api/v1/services?limit500resourceVersion0:dial tcp216.40.34.37:6443: connect: connection refused# DNS解析错误解决方法# 获取本机内网 IPLOCAL_IP192.168.108.32# 备份 hostscp/etc/hosts /etc/hosts.bak# 删除原有的错误解析 (如果有)sed-i/abner.com/d/etc/hostssed-i/^.*master$/d/etc/hosts# 添加正确的解析echo$LOCAL_IPmaster abner.com kubernetes kubernetes.default/etc/hosts# 验证cat/etc/hostsping-c1abner.comping-c1master清理残留环境# 1. 重置 kubeadmkubeadm reset-f# 2. 清理残留的 etcd 数据和 pki 证书 (因为证书里绑定了错误的 DNS/IP 组合建议重新生成)rm-rf/var/lib/etcd/*rm-rf/etc/kubernetes/pki/*# 注意保留 /etc/kubernetes/pki/etcd 如果不想重签 etcd 证书也可以但为了干净起见全删让脚本重生成最稳妥。# 这里我们只删 pki 目录下的内容保留目录结构rm-rf/etc/kubernetes/manifests/*rm-rf/var/lib/kubelet/*# 3. 重启 kubeletsystemctl restart kubelet查看node状态[rootmaster ~]# kubectl get nodes -o wideNAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME master Ready master 117m v1.19.5192.168.108.32noneCentOS Linux7(Core)3.10.0-1160.el7.x86_64 docker://19.3.114初始化worker节点获取join命令参数在master节点执行生成的token有效时间为2小时[rootmaster ~]# kubeadm token create --print-join-commandW022518:32:19.939401119982configset.go:348]WARNING: kubeadm cannot validate component configsforAPIgroups[kubelet.config.k8s.io kubeproxy.config.k8s.io]kubeadmjoinabner.com:6443--token19zdch.n9wx51hcnxasftfu --discovery-token-ca-cert-hash sha256:a466de0689ac189193277c99f61e8ac36b243c615324937276f430d460b59671针对所有worker节点执行# 只在 worker 节点执行# 替换 x.x.x.x 为 master 节点的内网 IP[rootnode1 ~]# export MASTER_IP192.168.108.32# 替换 apiserver.demo 为初始化 master 节点时所使用的 APISERVER_NAME[rootnode1 ~]# export APISERVER_NAMEabner.com[rootnode1 ~]# echo ${MASTER_IP} ${APISERVER_NAME} /etc/hosts[rootnode1 ~]# kubeadm join abner.com:6443 --token 19zdch.n9wx51hcnxasftfu --discovery-token-ca-cert-hash sha256:a466de0689ac189193277c99f61e8ac36b243c615324937276f430d460b59671初始化完成信息5检查初始化结果需要等待几分钟worker节点为Ready状态[rootmaster ~]# kubectl get nodeNAME STATUS ROLES AGE VERSION master Ready master 134m v1.19.5 node1 Readynone13m v1.19.5整合YAML文件准备因为yaml资源文件可以在k8s上创建应用pod所以需要提前创建yaml拉取到k8s服务器上在gitlab上新建YAML文件输入YAML文件内容apiVersion: apps/v1 kind: Deployment metadata: name: pipeline labels: app: pipeline spec: replicas:2selector: matchLabels: app: pipeline template: metadata: labels: app: pipeline spec: containers: - name: pipeline image:192.168.108.31:80/repo/pipeline:v4.0 imagePullPolicy: Always# 保持最新版本ports: - containerPort:8080--- apiVersion: v1 kind: Service metadata: labels: app: pipeline name: pipeline spec: selector: app: pipeline ports: - port:8081targetPort:8080type: NodePortharbor仓库对接在所有k8smasternode节点添加harbor仓库地址配置[rootmaster ~]# vim /etc/docker/daemon.json{insecure-registries:[192.168.108.31:80],registry-mirrors:[https://registry.cn-hangzhou.aliyuncs.com],exec-opts:[native.cgroupdriversystemd],log-driver:json-file,log-opts:{max-size:100m},storage-driver:overlay2,storage-opts:[overlay2.override_kernel_checktrue]}重启docker服务[rootmaster ~]# systemctl restart docker测试k8smasternode节点harbor登录[rootmaster ~]# docker login 192.168.108.31:80 -u admin -p harbor123WARNING!Using--passwordvia the CLI is insecure. Use --password-stdin. WARNING!Your password will be stored unencryptedin/root/.docker/config.json. Configure a credential helper to remove this warning. See https://docs.docker.com/engine/reference/commandline/login/#credentials-storeLogin SucceededYAML推送k8s服务器设置在master节点配置推送目标目录[rootmaster ~]# mkdir /usr/local/k8s[rootmaster ~]# chmod 777 /usr/local/k8s系统管理-系统配置点击测试成功后应用并保存在流水线中修改Publish Over SSH通知目标服务器步骤的Jenkinsfile的内容到gitlab服务器上修改Jenkinsfile内容保存配置并添加新标签tag否则识别不到新添加的yml文件先构建测试yaml文件推送查看构建日志构建失败原因Jenkins执行docker命令权限不足解决Jenkins节点执行以下命令cd/var/runchownroot:root docker.sockchmodorw docker.sock到master节点上查看已成功传递[rootmaster ~]# ls /usr/local/k8spipeline.yamlJenkins使用ssh无密码登录访问k8s的master节点因为希望使用ssh 用户名k8s地址 kubectl apply -f /usr/local/k8s/pipeline.yml创建资源会进行ssh 免交互把Jenkins中的公钥内容传递给k8s的master实现免密[rootjenkins ~]# docker exec -it jenkins bashjenkins51c548a67cdf:/$cd~ jenkins51c548a67cdf:~$ ssh-keygen-trsa Generating public/private rsa key pair. Enterfileinwhichto save the key(/var/jenkins_home/.ssh/id_rsa): Created directory/var/jenkins_home/.ssh.Enter passphrase(emptyforno passphrase): Enter same passphrase again: Your identification has been savedin/var/jenkins_home/.ssh/id_rsa Your public key has been savedin/var/jenkins_home/.ssh/id_rsa.pub The key fingerprint is: SHA256:0LH7tTXQ4j3nnD0wPBalGlIS4yKD67zJl91L/TkG4Y jenkins51c548a67cdf The keys randomart image is: ---[RSA3072]----|....||...o .o||.o..oo .oo.||.o......||.S. oB.||...o.Bo*o||...E.* o||..o. o o.||.|----[SHA256]-----jenkins51c548a67cdf:~$cd.ssh/ jenkins51c548a67cdf:~/.ssh$lsid_rsa id_rsa.pub复制公钥内容传递给masterjenkins51c548a67cdf:~/.ssh$catid_rsa.pub ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDuN33/QHO/MKpGnLT8n39qSGEHU7OPAaTZl9dQMWCFezM8WtLUJN7fUzOVixy1W2NKPSEckAMT1UTm8kLJM7yOAT/50bfVQvhM1LLjwPbIqrt4CeT/Hi0WhvBu9TzBZXuO88c/EOOO5mhBLQYOzHlGmu1gGlNORBYJRhyXQWUIPkMbH6jCYP1MCKLF85x5F/6bCsF65HhkLmf/i5XYDmYJeHMpeiHPhSfSgDtJFHd7Fq3XTfy4r9vrMthbpcpED3fpcqErXLY/m1V3QbJKC7szwsWjLzLeyIO2EYro7hIG9dEewlNeho57doL99NJkIL0jo27cqdvpBPwvP4niBKOratl/Wo6udeZBhBAi93nx/B5HE2ZjQ2iYYH6TF/USsWqs2YtFEv3yNaRmWGPg/VGojk99MpN2QgSCtw3Cumz4L3m/plPSmKErW39Xmbz3yFbUbNccgpnEhGvnZtQvk7m/unOLSFLI5MQ9P1k7iQ3fWRvNgXOTl8Yocjenkins51c548a67cdf到master节点上[rootmaster ~]# mkdir .ssh[rootmaster ~]# cd .ssh/[rootmaster .ssh]# vim authorized_keysssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDuN33/QHO/MKpGnLT8n39qSGEHU7OPAaTZl9dQMWCFezM8WtLUJN7fUzOVixy1W2NKPSEckAMT1UTm8kLJM7yOAT/50bfVQvhM1LLjwPbIqrt4CeT/Hi0WhvBu9TzBZXuO88c/EOOO5mhBLQYOzHlGmu1gGlNORBYJRhyXQWUIPkMbH6jCYP1MCKLF85x5F/6bCsF65HhkLmf/i5XYDmYJeHMpeiHPhSfSgDtJFHd7Fq3XTfy4r9vrMthbpcpED3fpcqErXLY/m1V3QbJKC7szwsWjLzLeyIO2EYro7hIG9dEewlNeho57doL99NJkIL0jo27cqdvpBPwvP4niBKOratl/Wo6udeZBhBAi93nx/B5HE2ZjQ2iYYH6TF/USsWqs2YtFEv3yNaRmWGPg/VGojk99MpN2QgSCtw3Cumz4L3m/plPSmKErW39Xmbz3yFbUbNccgpnEhGvnZtQvk7m/unOLSFLI5MQ9P1k7iQ3fWRvNgXOTl8Yocjenkins51c548a67cdfmaster重启sshd服务[rootmaster .ssh]# systemctl restart sshd在Jenkins容器中测试免密执行jenkins51c548a67cdf:~/.ssh$sshroot192.168.108.32ls/ bin boot dev etc home lib lib64 media mnt opt proc root run sbin srv sys tmp usr varJenkins中设置YAML部署在流水线语法中输入注意执行的是绝对路径sshroot192.168.108.32 kubectl apply-f/usr/local/k8s/pipeline.yaml在gitlab中增加步骤增加步骤内容stage(远程执行kubectl命令){steps{shssh root192.168.108.32 kubectl apply -f /usr/local/k8s/pipeline.yaml}}因为拉取的项目标签是4.0所以pipeline.yml拉取镜像的标签也要同步变更首页内容变更路径src/main/java/com/guo/demo/controller/Test/Controller.javagitlab中把原来的v4.0标签删除重新创建命名依然写v4.0立即构建在master上查看pod资源[rootmaster .ssh]# kubectl get pod -o wideNAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES pipeline-6b4965b559-657dm1/1 Running0112s10.100.166.129 node1nonenonepipeline-6b4965b559-mbj7p1/1 Running0112s10.100.166.130 node1nonenone[rootmaster .ssh]# kubectl get svcNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)AGE kubernetes ClusterIP10.96.0.1none443/TCP 18h pipeline NodePort10.96.53.129none8081:31463/TCP 2m9s在node1上查看下载的pipeline镜像[rootnode1 ~]# docker imagesREPOSITORY TAG IMAGE ID CREATED SIZE192.168.108.31:80/repo/pipeline v4.0 1332856cac9d2minutes ago 543MB[rootnode1 ~]# docker ps -a | grep pipeline29dc0d1c1fc5192.168.108.31:80/repo/pipeline/bin/sh -c java -j…2minutes ago Up2minutes k8s_pipeline_pipeline-6b4965b559-mbj7p_default_90e329ee-1d45-4c20-90e1-9a5ec26d30fa_0 81dd1211dbe2192.168.108.31:80/repo/pipeline/bin/sh -c java -j…2minutes ago Up2minutes k8s_pipeline_pipeline-6b4965b559-657dm_default_cef13886-b974-426d-ba87-bab6e11c80f8_0 bee3825f8892 registry.aliyuncs.com/k8sxio/pause:3.2/pause2minutes ago Up2minutes k8s_POD_pipeline-6b4965b559-mbj7p_default_90e329ee-1d45-4c20-90e1-9a5ec26d30fa_0 6d6bfdab414c registry.aliyuncs.com/k8sxio/pause:3.2/pause2minutes ago Up2minutes k8s_POD_pipeline-6b4965b559-657dm_default_cef13886-b974-426d-ba87-bab6e11c80f8_0打开node地址查看网页http://192.168.108.33:31463/test查看钉钉通知
