目录1.概述1.1redis-shake软件概述1.1.1redisshake基本介绍1.1.2redisshake使用背景1.1.3redisshake功能说明1.1.4redisshake特性概览1.2测试环境概述2.环境准备2.1下载redis2.2安装redis2.3配置redis3.安装配置redisshake3.1下载redisshake3.2安装redisshake3.3配置redisshake4.同步测试4.1源端创建测试数据4.2执行redis数据迁移4.3验证迁移结果1.概述1.1redis-shake软件概述1.1.1redisshake基本介绍RedisShake是基于redis-port基础上进行改进的是一款开源的Redis迁移工具支持Cluster集群的在线迁移与离线迁移备份文件导入。数据可平滑迁移当部署在其他云厂商Redis服务上的Cluster集群数据由于SYNC、PSYNC命令被云厂商禁用无法在线迁移时可以选择离线迁移。1.1.2redisshake使用背景RedisShake是基于redis-port基础上进行改进的是一款开源的Redis迁移工具支持Cluster集群的在线迁移与离线迁移备份文件导入。数据可平滑迁移当部署在其他云厂商Redis服务上的Cluster集群数据由于SYNC、PSYNC命令被云厂商禁用无法在线迁移时可以选择离线迁移。1.1.3redisshake功能说明RedisShake是基于redis-port基础上进行改进的是一款开源的Redis迁移工具支持Cluster集群的在线迁移与离线迁移备份文件导入。数据可平滑迁移当部署在其他云厂商Redis服务上的Cluster集群数据由于SYNC、PSYNC命令被云厂商禁用无法在线迁移时可以选择离线迁移。1.1.4redisshake特性概览高性能全量同步阶段并发执行增量同步阶段异步执行能够达到毫秒级别延迟取决于网络延迟。同时我们还对大key同步进行分批拉取优化同步性能。在 Redis 5.0、Redis 6.0 和 Redis 7.0 上测试支持使用lua自定义过滤规则支持大实例迁移支持restore模式和sync模式支持阿里云 Redis 和 ElastiCache监控体系用户可以通过我们提供的restful拉取metric来对redis-shake进行实时监控curl 127.0.0.1:9320/metric。数据校验如何校验同步的正确性可以采用我们开源的redis-full-check。支持版本支持2.8-5.0版本的同步此外还支持codis支持云下到云上云上到云上云上到云下阿里云目前支持主从版其他云到阿里云等链路帮助用户灵活构建混合云场景。断点续传。支持断开后按offset恢复降低因主备切换、网络抖动造成链路断开重新同步拉取全量的性能影响。1.2测试环境概述在两台机器创建两个redis服务一个redis6另一个redis7在其中一个添加数据配置redis-shake进行数据同步测试序号机器名ip地址操作系统版本基础环境1redis610.9.254.31RockyLinux8.7redis6、redis-shake2redis710.9.254.32RockyLinux8.7redis72.环境准备2.1下载redishttps://download.redis.io/releases/https://download.redis.io/releases/redis-6.0.0.tar.gzhttps://download.redis.io/releases/redis-7.0.0.tar.gz[rootredis6 ~]# wget https://download.redis.io/releases/redis-6.0.0.tar.gz[rootredis6 ~]# ll-rw-r--r-- 1 root root 2204177 Jun 27 2020 redis-6.0.0.tar.gz[rootredis7 ~]# wget https://download.redis.io/releases/redis-7.0.0.tar.gz[rootredis7 ~]# ll-rw-r--r-- 1 root root 2943054 Apr 27 2022 redis-7.0.0.tar.gz2.2安装redis安装redis6相同方法在另一台安装redis7[rootredis6 ~]# yum -y install wget gcc tcl makeUpgraded:cpp-8.5.0-22.el8_10.x86_64 gcc-8.5.0-22.el8_10.x86_64gcc-c-8.5.0-22.el8_10.x86_64 libgcc-8.5.0-22.el8_10.x86_64libgomp-8.5.0-22.el8_10.x86_64 libstdc-8.5.0-22.el8_10.x86_64libstdc-devel-8.5.0-22.el8_10.x86_64 wget-1.19.5-12.el8_10.x86_64Installed:make-1:4.2.1-11.el8.x86_64 tcl-1:8.6.8-2.el8.x86_64Complete![rootredis6 ~]# tar -zxvf redis-6.0.0.tar.gz[rootredis6 ~]# cd redis-6.0.0[rootredis6 ~/redis-6.0.0]# makeLINK redis-serverINSTALL redis-sentinelCC redis-cli.oLINK redis-cliCC redis-benchmark.oLINK redis-benchmarkINSTALL redis-check-rdbINSTALL redis-check-aofHint: Its a good idea to run make test ;)make[1]: Leaving directory /root/redis-6.0.0/src[rootredis6 ~/redis-6.0.0]# cd /root/redis-6.0.0/src[rootredis6 ~/redis-6.0.0/src]# make installCC Makefile.depHint: Its a good idea to run make test ;)INSTALL installINSTALL installINSTALL installINSTALL installINSTALL install2.3配置redis[rootredis6 ~]# mkdir -p /usr/local/redis/etc/[rootredis6 ~]# mkdir -p /usr/local/redis/bin[rootredis6 ~]# cd /root/redis-6.0.0/[rootredis6 ~/redis-6.0.0]# cp redis.conf /usr/local/redis/etc/[rootredis6 ~/redis-6.0.0]# cd /root/redis-6.0.0/src[rootredis6 ~/redis-6.0.0/src]# cp mkreleasehdr.sh redis-benchmark redis-check-aof redis-check-rdb redis-cli redis-server /usr/local/redis/bin/[rootredis6 ~/redis-6.0.0/src]# vim /usr/local/redis/etc/redis.confbind 0.0.0.0daemonize yesrequirepass P88w0rd[rootredis6 ~/redis-6.0.0/src]# vim /etc/profileexport REDIS_HOME/usr/local/redisexport PATH$PATH:$REDIS_HOME/bin/[rootredis6 ~/redis-6.0.0/src]# source /etc/profile[rootredis6 ~/redis-6.0.0/src]# redis-server /usr/local/redis/etc/redis.conf15019:C 05 Dec 2024 17:23:23.593 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo15019:C 05 Dec 2024 17:23:23.593 # Redis version6.0.0, bits64, commit00000000, modified0, pid15019, just started15019:C 05 Dec 2024 17:23:23.593 # Configuration loaded[rootredis6 ~/redis-6.0.0/src]# netstat -ntlpActive Internet connections (only servers)Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program nametcp 0 0 0.0.0.0:6379 0.0.0.0:* LISTEN 15020/redis-servertcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 949/sshdtcp6 0 0 :::22 :::* LISTEN 949/sshd[rootredis6 ~/redis-6.0.0/src]# redis-server --versionRedis server v6.0.0 sha00000000:0 mallocjemalloc-5.1.0 bits64 build0设置脚本控制启停[rootredis6 ~]# vim /usr/local/redis/bin/redis-start#/bin/bash/usr/local/redis/bin/redis-server /usr/local/redis/etc/redis.conf[rootredis6 ~]# vim /usr/local/redis/bin/redis-stop#!/bin/bashecho redis-server will stopredis_pid$(ps -ef |grep -v grep | egrep /usr/local/redis/bin/redis-server | awk {printf $2 });if [ $redis_pid ! ]; thenkill -9 $redis_pidecho redis-server stopelseecho can not find redis-serverfiexit 13.安装配置redisshake3.1下载redisshakehttps://github.com/tair-opensource/RedisShake/releases/tag/v4.2.2https://github.com/tair-opensource/RedisShake/releases/download/v4.2.2/redis-shake-linux-amd64.tar.gz[rootredis6 ~]# wget https://github.com/tair-opensource/RedisShake/releases/download/v4.2.2/redis-shake-linux-amd64.tar.gz[rootredis6 ~]# ll-rw-r--r-- 1 root root 6636344 Dec 5 13:22 redis-shake-linux-amd64.tar.gz3.2安装redisshake[rootredis6 ~]# tar -zxvf redis-shake-linux-amd64.tar.gz./redis-shake./shake.toml[rootredisshake ~]# ll-rwxr-xr-x 1 1001 docker 11739313 Nov 19 15:16 redis-shake-rw-r--r-- 1 root root 6636344 Dec 5 13:22 redis-shake-linux-amd64.tar.gz-rw-r--r-- 1 1001 docker 6002 Nov 19 15:15 shake.toml[rootredis6 ~]# mv redis-shake /usr/local/bin/[rootredis6 ~]# ll /usr/local/bin/-rwxr-xr-x 1 1001 127 11739313 Nov 19 15:16 redis-shake3.3配置redisshake[rootredis6 ~]# cp shake.toml shake.toml.bak[rootredis6 ~]# vim shake.toml[sync_reader]address 10.9.254.31:6379password P88w0rd[redis_writer]address 10.9.254.32:6379password P88w0rd4.同步测试4.1源端创建测试数据[rootredis6 ~]# redis-cli -h 10.9.254.31 -p 6379 -a P88w0rdWarning: Using a password with -a or -u option on the command line interface may not be safe.10.9.254.31:6379 pingPONG10.9.254.31:6379 SET key1 value1OK4.2执行redis数据迁移[rootredis6 ~]# redis-shake ./shake.toml2024-12-09 13:55:40 INF load config from file: ./shake.toml2024-12-09 13:55:40 INF log_level: [info], log_file: [/root/data/shake.log]2024-12-09 13:55:40 INF changed work dir. dir[/root/data]2024-12-09 13:55:40 INF GOMAXPROCS defaults to the value of runtime.NumCPU [2]2024-12-09 13:55:40 INF not set pprof port2024-12-09 13:55:40 INF create SyncStandaloneReader: 10.9.254.31:63792024-12-09 13:55:40 INF create RedisStandaloneWriter: 10.9.254.32:63792024-12-09 13:55:40 INF start syncing...2024-12-09 13:55:40 INF [reader_10.9.254.31_6379] source db is not doing bgsave! continue.2024-12-09 13:55:45 INF read_count[1], read_ops[0.00], write_count[1], write_ops[0.00], syncing aof, diff[0]2024-12-09 13:55:50 INF read_count[1], read_ops[0.20], write_count[1], write_ops[0.20], syncing aof, diff[0]2024-12-09 13:55:55 INF read_count[1], read_ops[0.00], write_count[1], write_ops[0.00], syncing aof, diff[0]2024-12-09 13:56:00 INF read_count[1], read_ops[0.00], write_count[1], write_ops[0.00], syncing aof, diff[0]2024-12-09 13:56:05 INF read_count[1], read_ops[0.00], write_count[1], write_ops[0.00], syncing aof, diff[0]4.3验证迁移结果[rootredis7 ~]# redis-cli -h 10.9.254.32 -p 6379 -a P88w0rdWarning: Using a password with -a or -u option on the command line interface may not be safe.10.9.254.32:6379 pingPONG10.9.254.32:6379 GET key1value1手动在源redis中添加数据再次验证[rootredis6 ~]# redis-cli -h 10.9.254.31 -p 6379 -a P88w0rdWarning: Using a password with -a or -u option on the command line interface may not be safe.10.9.254.31:6379 pingPONG10.9.254.31:6379 SET key2 value2OK[rootredis7 ~]# redis-cli -h 10.9.254.32 -p 6379 -a P88w0rdWarning: Using a password with -a or -u option on the command line interface may not be safe.10.9.254.32:6379 pingPONG10.9.254.32:6379 GET key2value2
redis-shake-在rockylinux8上数据同步测试
目录1.概述1.1redis-shake软件概述1.1.1redisshake基本介绍1.1.2redisshake使用背景1.1.3redisshake功能说明1.1.4redisshake特性概览1.2测试环境概述2.环境准备2.1下载redis2.2安装redis2.3配置redis3.安装配置redisshake3.1下载redisshake3.2安装redisshake3.3配置redisshake4.同步测试4.1源端创建测试数据4.2执行redis数据迁移4.3验证迁移结果1.概述1.1redis-shake软件概述1.1.1redisshake基本介绍RedisShake是基于redis-port基础上进行改进的是一款开源的Redis迁移工具支持Cluster集群的在线迁移与离线迁移备份文件导入。数据可平滑迁移当部署在其他云厂商Redis服务上的Cluster集群数据由于SYNC、PSYNC命令被云厂商禁用无法在线迁移时可以选择离线迁移。1.1.2redisshake使用背景RedisShake是基于redis-port基础上进行改进的是一款开源的Redis迁移工具支持Cluster集群的在线迁移与离线迁移备份文件导入。数据可平滑迁移当部署在其他云厂商Redis服务上的Cluster集群数据由于SYNC、PSYNC命令被云厂商禁用无法在线迁移时可以选择离线迁移。1.1.3redisshake功能说明RedisShake是基于redis-port基础上进行改进的是一款开源的Redis迁移工具支持Cluster集群的在线迁移与离线迁移备份文件导入。数据可平滑迁移当部署在其他云厂商Redis服务上的Cluster集群数据由于SYNC、PSYNC命令被云厂商禁用无法在线迁移时可以选择离线迁移。1.1.4redisshake特性概览高性能全量同步阶段并发执行增量同步阶段异步执行能够达到毫秒级别延迟取决于网络延迟。同时我们还对大key同步进行分批拉取优化同步性能。在 Redis 5.0、Redis 6.0 和 Redis 7.0 上测试支持使用lua自定义过滤规则支持大实例迁移支持restore模式和sync模式支持阿里云 Redis 和 ElastiCache监控体系用户可以通过我们提供的restful拉取metric来对redis-shake进行实时监控curl 127.0.0.1:9320/metric。数据校验如何校验同步的正确性可以采用我们开源的redis-full-check。支持版本支持2.8-5.0版本的同步此外还支持codis支持云下到云上云上到云上云上到云下阿里云目前支持主从版其他云到阿里云等链路帮助用户灵活构建混合云场景。断点续传。支持断开后按offset恢复降低因主备切换、网络抖动造成链路断开重新同步拉取全量的性能影响。1.2测试环境概述在两台机器创建两个redis服务一个redis6另一个redis7在其中一个添加数据配置redis-shake进行数据同步测试序号机器名ip地址操作系统版本基础环境1redis610.9.254.31RockyLinux8.7redis6、redis-shake2redis710.9.254.32RockyLinux8.7redis72.环境准备2.1下载redishttps://download.redis.io/releases/https://download.redis.io/releases/redis-6.0.0.tar.gzhttps://download.redis.io/releases/redis-7.0.0.tar.gz[rootredis6 ~]# wget https://download.redis.io/releases/redis-6.0.0.tar.gz[rootredis6 ~]# ll-rw-r--r-- 1 root root 2204177 Jun 27 2020 redis-6.0.0.tar.gz[rootredis7 ~]# wget https://download.redis.io/releases/redis-7.0.0.tar.gz[rootredis7 ~]# ll-rw-r--r-- 1 root root 2943054 Apr 27 2022 redis-7.0.0.tar.gz2.2安装redis安装redis6相同方法在另一台安装redis7[rootredis6 ~]# yum -y install wget gcc tcl makeUpgraded:cpp-8.5.0-22.el8_10.x86_64 gcc-8.5.0-22.el8_10.x86_64gcc-c-8.5.0-22.el8_10.x86_64 libgcc-8.5.0-22.el8_10.x86_64libgomp-8.5.0-22.el8_10.x86_64 libstdc-8.5.0-22.el8_10.x86_64libstdc-devel-8.5.0-22.el8_10.x86_64 wget-1.19.5-12.el8_10.x86_64Installed:make-1:4.2.1-11.el8.x86_64 tcl-1:8.6.8-2.el8.x86_64Complete![rootredis6 ~]# tar -zxvf redis-6.0.0.tar.gz[rootredis6 ~]# cd redis-6.0.0[rootredis6 ~/redis-6.0.0]# makeLINK redis-serverINSTALL redis-sentinelCC redis-cli.oLINK redis-cliCC redis-benchmark.oLINK redis-benchmarkINSTALL redis-check-rdbINSTALL redis-check-aofHint: Its a good idea to run make test ;)make[1]: Leaving directory /root/redis-6.0.0/src[rootredis6 ~/redis-6.0.0]# cd /root/redis-6.0.0/src[rootredis6 ~/redis-6.0.0/src]# make installCC Makefile.depHint: Its a good idea to run make test ;)INSTALL installINSTALL installINSTALL installINSTALL installINSTALL install2.3配置redis[rootredis6 ~]# mkdir -p /usr/local/redis/etc/[rootredis6 ~]# mkdir -p /usr/local/redis/bin[rootredis6 ~]# cd /root/redis-6.0.0/[rootredis6 ~/redis-6.0.0]# cp redis.conf /usr/local/redis/etc/[rootredis6 ~/redis-6.0.0]# cd /root/redis-6.0.0/src[rootredis6 ~/redis-6.0.0/src]# cp mkreleasehdr.sh redis-benchmark redis-check-aof redis-check-rdb redis-cli redis-server /usr/local/redis/bin/[rootredis6 ~/redis-6.0.0/src]# vim /usr/local/redis/etc/redis.confbind 0.0.0.0daemonize yesrequirepass P88w0rd[rootredis6 ~/redis-6.0.0/src]# vim /etc/profileexport REDIS_HOME/usr/local/redisexport PATH$PATH:$REDIS_HOME/bin/[rootredis6 ~/redis-6.0.0/src]# source /etc/profile[rootredis6 ~/redis-6.0.0/src]# redis-server /usr/local/redis/etc/redis.conf15019:C 05 Dec 2024 17:23:23.593 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo15019:C 05 Dec 2024 17:23:23.593 # Redis version6.0.0, bits64, commit00000000, modified0, pid15019, just started15019:C 05 Dec 2024 17:23:23.593 # Configuration loaded[rootredis6 ~/redis-6.0.0/src]# netstat -ntlpActive Internet connections (only servers)Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program nametcp 0 0 0.0.0.0:6379 0.0.0.0:* LISTEN 15020/redis-servertcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 949/sshdtcp6 0 0 :::22 :::* LISTEN 949/sshd[rootredis6 ~/redis-6.0.0/src]# redis-server --versionRedis server v6.0.0 sha00000000:0 mallocjemalloc-5.1.0 bits64 build0设置脚本控制启停[rootredis6 ~]# vim /usr/local/redis/bin/redis-start#/bin/bash/usr/local/redis/bin/redis-server /usr/local/redis/etc/redis.conf[rootredis6 ~]# vim /usr/local/redis/bin/redis-stop#!/bin/bashecho redis-server will stopredis_pid$(ps -ef |grep -v grep | egrep /usr/local/redis/bin/redis-server | awk {printf $2 });if [ $redis_pid ! ]; thenkill -9 $redis_pidecho redis-server stopelseecho can not find redis-serverfiexit 13.安装配置redisshake3.1下载redisshakehttps://github.com/tair-opensource/RedisShake/releases/tag/v4.2.2https://github.com/tair-opensource/RedisShake/releases/download/v4.2.2/redis-shake-linux-amd64.tar.gz[rootredis6 ~]# wget https://github.com/tair-opensource/RedisShake/releases/download/v4.2.2/redis-shake-linux-amd64.tar.gz[rootredis6 ~]# ll-rw-r--r-- 1 root root 6636344 Dec 5 13:22 redis-shake-linux-amd64.tar.gz3.2安装redisshake[rootredis6 ~]# tar -zxvf redis-shake-linux-amd64.tar.gz./redis-shake./shake.toml[rootredisshake ~]# ll-rwxr-xr-x 1 1001 docker 11739313 Nov 19 15:16 redis-shake-rw-r--r-- 1 root root 6636344 Dec 5 13:22 redis-shake-linux-amd64.tar.gz-rw-r--r-- 1 1001 docker 6002 Nov 19 15:15 shake.toml[rootredis6 ~]# mv redis-shake /usr/local/bin/[rootredis6 ~]# ll /usr/local/bin/-rwxr-xr-x 1 1001 127 11739313 Nov 19 15:16 redis-shake3.3配置redisshake[rootredis6 ~]# cp shake.toml shake.toml.bak[rootredis6 ~]# vim shake.toml[sync_reader]address 10.9.254.31:6379password P88w0rd[redis_writer]address 10.9.254.32:6379password P88w0rd4.同步测试4.1源端创建测试数据[rootredis6 ~]# redis-cli -h 10.9.254.31 -p 6379 -a P88w0rdWarning: Using a password with -a or -u option on the command line interface may not be safe.10.9.254.31:6379 pingPONG10.9.254.31:6379 SET key1 value1OK4.2执行redis数据迁移[rootredis6 ~]# redis-shake ./shake.toml2024-12-09 13:55:40 INF load config from file: ./shake.toml2024-12-09 13:55:40 INF log_level: [info], log_file: [/root/data/shake.log]2024-12-09 13:55:40 INF changed work dir. dir[/root/data]2024-12-09 13:55:40 INF GOMAXPROCS defaults to the value of runtime.NumCPU [2]2024-12-09 13:55:40 INF not set pprof port2024-12-09 13:55:40 INF create SyncStandaloneReader: 10.9.254.31:63792024-12-09 13:55:40 INF create RedisStandaloneWriter: 10.9.254.32:63792024-12-09 13:55:40 INF start syncing...2024-12-09 13:55:40 INF [reader_10.9.254.31_6379] source db is not doing bgsave! continue.2024-12-09 13:55:45 INF read_count[1], read_ops[0.00], write_count[1], write_ops[0.00], syncing aof, diff[0]2024-12-09 13:55:50 INF read_count[1], read_ops[0.20], write_count[1], write_ops[0.20], syncing aof, diff[0]2024-12-09 13:55:55 INF read_count[1], read_ops[0.00], write_count[1], write_ops[0.00], syncing aof, diff[0]2024-12-09 13:56:00 INF read_count[1], read_ops[0.00], write_count[1], write_ops[0.00], syncing aof, diff[0]2024-12-09 13:56:05 INF read_count[1], read_ops[0.00], write_count[1], write_ops[0.00], syncing aof, diff[0]4.3验证迁移结果[rootredis7 ~]# redis-cli -h 10.9.254.32 -p 6379 -a P88w0rdWarning: Using a password with -a or -u option on the command line interface may not be safe.10.9.254.32:6379 pingPONG10.9.254.32:6379 GET key1value1手动在源redis中添加数据再次验证[rootredis6 ~]# redis-cli -h 10.9.254.31 -p 6379 -a P88w0rdWarning: Using a password with -a or -u option on the command line interface may not be safe.10.9.254.31:6379 pingPONG10.9.254.31:6379 SET key2 value2OK[rootredis7 ~]# redis-cli -h 10.9.254.32 -p 6379 -a P88w0rdWarning: Using a password with -a or -u option on the command line interface may not be safe.10.9.254.32:6379 pingPONG10.9.254.32:6379 GET key2value2
