Redis Sentinel是Redis的高可用方案。是Redis 2.8中正式引入的。
在之前的主从复制方案中，如果主节点出现问题，需要手动将一个从节点升级为主节点，然后将其它从节点指向新的主节点，并且需要修改应用方主节点的地址。整个过程都需要人工干预。
下面通过日志具体看看Sentinel的切换流程。
Sentinel的切换流程
集群拓扑图如下。
角色 IP 端口 runID
主节点 127.0.0.1 6379
从节点-1 127.0.0.1 6380
从节点-2 127.0.0.1 6381
Sentinel-1 127.0.0.1 26379 d4424b8684977767be4f5abd1e364153fbb0adbd
Sentinel-2 127.0.0.1 26380 18311edfbfb7bf89fe4b67d08ef432053db62fff
Sentinel-3 127.0.0.1 26381 3e9eb1aa9378d89cfe04fe21bf4a05a901747fa8
kill -9 将主节点进程杀死。
1. 最先反应的是从节点。
其会马上输出如下信息。
28244:S 08 Oct 16:03:34.184 # Connection with master lost. 34.184 * Caching the disconnected master state. 34.548 * Connecting to MASTER 127.0.0.1:6379 34.548 * MASTER <-> SLAVE sync started 34.548 # Error condition on socket for SYNC: Connection refused 35.556 * Connecting to MASTER 35.556 * MASTER <-> SLAVE sync started ...
2. Sentinel的日志30s后才有输出，这个与“sentinel down-after-milliseconds mymaster 30000”的设置有关。
下面，依次贴出哨兵各个节点及slave的日志输出。
Sentinel-1
28087:X 04:04.277 # +sdown master mymaster 0.1 04.379 # +new-epoch 1 04.385 # +vote-for-leader 18311edfbfb7bf89fe4b67d08ef432053db62fff 05.388 # +odown master mymaster 6379 #quorum 3/2 05.388 # Next failover delay: I will not start a failover before Mon Oct 8 10:04 2018 05.631 # +config-update-from sentinel 18311edfbfb7bf89fe4b67d08ef432053db62fff 26380 @ mymaster 05.631 # +switch-master mymaster 6379 6381 05.631 * +slave slave 6380 6380 @ mymaster 6379 @ mymaster 35.656 # +sdown slave 6381
Sentinel-2
28163:X 04.289 # +sdown master mymaster 04.366 # +odown master mymaster 04.366 # +try-failover master mymaster 04.373 # +vote-04.385 # 3e9eb1aa9378d89cfe04fe21bf4a05a901747fa8 voted for 18311edfbfb7bf89fe4b67d08ef432053db62fff 04.385 # d4424b8684977767be4f5abd1e364153fbb0adbd voted 04.450 # +elected-leader master mymaster 04.450 # +failover-state-select-slave master mymaster 04.528 # +selected-slave slave 6381 6381 @ mymaster 04.528 * +failover-state-send-slaveof-noone slave 04.586 * +failover-state-wait-promotion slave 05.543 # +promoted-slave slave 05.543 # +failover-state-reconf-slaves master mymaster 05.629 * +slave-reconf-sent slave 06.554 # -odown master mymaster 06.555 * +slave-reconf-inprog slave 06.555 * +slave-reconf-done slave 06.606 # +failover-end master mymaster 06.606 # +06.606 * +slave slave 36.687 # +sdown slave 6381
Sentinel-3
28234:X 04.288 # +sdown master mymaster 04.378 # +04.385 # +odown master mymaster 2/04.385 # Next failover delay: I will not start a failover before Mon Oct 05.630 # +config-update-05.630 # +05.630 * +slave slave 35.709 # +sdown slave 6381
slave2
04.762 * MASTER <->04.762 # Error condition on socket 05.630 * SLAVE OF 6381 enabled (user request from 'id=6 addr=127.0.0.1:43880 fd=12 name= age=148 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=224 qbuf-free= 32544 obl=81 oll=0 omem=0 events=r cmd=slaveof)28244:S 08 Oct 16:04:05.636 # CONFIG REWRITE executed with success. 05.770 * Connecting to MASTER 05.770 * MASTER <->05.770 * Non blocking connect for SYNC fired the event. 05.770 * Master replied to PING,replication can continue... 05.770 * Trying a partial resynchronization (request b95802ca8afd97c578b355a5838d219681d0af27:24302). 05.770 * Successful partial resynchronization with master. 05.770 # Master replication ID changed to a4022bb5c361353a4773fd460cec5cdcc5c02031 05.770 * MASTER <-> SLAVE sync: Master accepted a Partial Resynchronization.
slave3
28253:S 03.655 * MASTER <->03.655 # Error condition on socket 28253:M 04.586 # Setting secondary replication ID to b95802ca8afd97c578b355a5838d219681d0af27,valid up to offset: 24302. New replication ID is a4022bb5c361353a4773fd460cec5cdc c5c0203128253:M 04.586 * Discarding previously cached master state. 04.586 * MASTER MODE enabled (user request id=9 addr=127.0.0.1:49316 fd=8 name=sentinel-18311edf-cmd age=137 idle=0 flags=x db=0 sub=0 psub=0 multi=3 qbuf=0 qbuf- free=32768 obl=36 oll=0 events=r cmd=exec)28253:M 08 Oct 16:04:04.593 # CONFIG REWRITE executed with success. 05.770 * Slave 6380 asks synchronization 05.770 * Partial resynchronization request 6380 accepted. Sending 156 bytes of backlog starting from offset 24302.
结合上面的日志，可以看到，
各个Sentinel节点都判断127.0.0.1 6379为主观下线（Subjectively Down，缩写为sdown）。
6379
达到quorum的设置，Sentinel-2判断其为客观下线（Objectively Down，缩写为odown）。结合其它两个Sentinel节点的日志，可以看到，Sentinel-2最先判定其客观下线。接下来，会进行Sentinel的领导者选举。一般来说，谁先完成客观下线的判定，谁就是领导者，只有Sentinel领导者才能进行failover。
6379
寻找合适的slave作为master
6379
+failover-state-select-slave <instance details> -- New failover state is select-slave: we are trying to find a suitable slave for promotion.
将127.0.0.1 6381设置为新主
6379
+selected-slave <instance details> -- We found the specified good slave to promote.
命令6381节点执行slaveof no one，使其成为主节点
6379
+failover-state-send-slaveof-noone <instance details> -- We are trying to reconfigure the promoted slave as master,waiting for it to switch.
等待6381节点升级为主节点
6379
确认6381节点已经升级为主节点
6379
再来看看16:04:04.528到16:04:05.543这个时间段slave3的日志输出。可以看到，其开启了MASTER模式，且重写了配置文件。
a4022bb5c361353a4773fd460cec5cdcc5c02031
28253:M id=9 addr=127.0.0.1:49316 fd=8 name=sentinel-18311edf-cmd age=137 idle=0 flags=x db=0 sub=0 psub=0 multi=3 qbuf=0 qbuf-free=)
28253:M 08 Oct 16:04:04.593 # CONFIG REWRITE executed with success.
failover进入重新配置从节点阶段
6379
命令6380节点复制新的主节点
6379
+slave-reconf-sent <instance details> -- The leader sentinel sent the SLAVEOF command to this instance in order to reconfigure it for the new slave.
看看这个时间点slave2的日志输出，基本吻合。其进行的是增量同步。
id=6 addr=127.0.0.1:43880 fd=12 name= age=148 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=224 qbuf-free=)
28244:S 08 Oct 16:04:05.636 # CONFIG REWRITE executed with success. 05.770 * MASTER <-> SLAVE sync: Master accepted a Partial Resynchronization.
同时，在这个时间点，sentinel也有日志输出，以sentinel1为例。从日志中，可以看到，在这个时间点它会更改配置信息。
6381
switch-master <master name> <oldip> <oldport> <newip> <newport> -- The master new IP and address is the specified one after a configuration change. This is the message most external users are interested in.
同步过程尚未完成。
6379
+slave-reconf-inprog <instance details> -- The slave being reconfigured showed to be a slave of the new master ip:port pair,but the synchronization process is not yet complete.
主从同步完成。
6379
+slave-reconf-done <instance details> -- The slave is now synchronized with the new master.
failover切换完成。
6379
failover成功后，发布主节点的切换消息
6381
关联新主节点的slave信息，需要注意的是，原来的主节点会作为新主节点的slave。
6381
+slave <instance details> -- A new slave was detected and attached.
过了30s后，判定原来的主节点主观下线。
6381
综合来看，Sentinel进行failover的流程如下
1. 每隔1秒，每个Sentinel节点会向主节点、从节点、其余Sentinel节点发送一条ping命令做一次心跳检测，来确认这些节点当前是否可达。当这些节点超过down-after-milliseconds没有进行有效回复，Sentinel节点就会判定该节点为主观下线。
2. 如果被判定为主观下线的节点是主节点，该Sentinel节点会通过sentinel is master-down-by-addr命令向其他Sentinel节点询问对主节点的判断，当超过<quorum>个数，Sentinel节点会判定该节点为客观下线。如果从节点、Sentinel节点被判定为主观下线，并不会进行后续的故障切换操作。
3. 对Sentinel进行领导者选举，由其来进行后续的故障切换（failover）工作。选举算法基于Raft。
4. Sentinel领导者节点开始进行故障切换。
5. 选择合适的从节点作为新主节点。
6. Sentinel领导者节点对上一步选出来的从节点执行slaveof no one命令让其成为主节点。
7. 向剩余的从节点发送命令，让它们成为新主节点的从节点，复制规则和parallel-syncs参数有关。
8. 将原来的主节点更新为从节点，并将其纳入到Sentinel的管理，让其恢复后去复制新的主节点。
Sentinel的领导者选举流程。
Sentinel的领导者选举基于Raft协议。
1. 每个在线的Sentinel节点都有资格成为领导者，当它确认主节点主观下线时候，会向其他Sentinel节点发送sentinel is-master-down-by-addr命令，要求将自己设置为领导者。
2. 收到命令的Sentinel节点，如果没有同意过其他Sentinel节点的sentinel is-master-down-by-addr命令，将同意该请求，否则拒绝。
3. 如果该Sentinel节点发现自己的票数已经大于等于max（quorum，num（sentinels）/2+1），那么它将成为领导者。
新主节点的选择流程。
1. 删除所有已经处于下线或断线状态的从节点。
2. 删除最近5秒没有回复过领导者Sentinel的INFO命令的从节点。
3. 删除所有与已下线主节点连接断开超过down-after-milliseconds*10毫秒的从节点。
4. 选择优先级最高的从节点。
5. 选择复制偏移量最大的从节点。
6. 选择runid最小的从节点。
三个定时监控任务
1. 每隔10秒，每个Sentinel节点会向主节点和从节点发送info命令获取最新的拓扑结构。其作用如下：
1> 通过向主节点执行info命令，获取从节点的信息，这也是为什么Sentinel节点不需要显式配置监控从节点。
2> 当有新的从节点加入时可立刻感知出来。
3> 节点不可达或者故障切换后，可通过info命令实时更新节点拓扑信息。
2. 每隔2秒，每个Sentinel节点会向Redis数据节点的__sentinel__：hello频道上发送该Sentinel节点对于主节点的判断以及当前Sentinel节点的信息，同时每个Sentinel节点也会订阅该频道，来了解其它Sentinel节点以及它们对主节点的判断。其作用如下：
1> 发现新的Sentinel节点：通过订阅主节点的__sentinel__：hello了解其它Sentinel节点信息，如果是新加入的Sentinel节点，将该Sentinel节点信息保存起来，并与该Sentinel节点创建连接。
2> Sentinel节点之间交换主节点的状态，作为后面客观下线以及领导者选举的依据。
3. 每隔1秒，每个Sentinel节点会向主节点、从节点、其余Sentinel节点发送一条ping命令做一次心跳检测，来确认这些节点当前是否可达。这个定时任务是节点失败判定的重要依据。
Sentinel的相关参数
# bind 127.0.0.1 192.168.1.1 # protected-mode no port 26379 # sentinel announce-ip <ip> # sentinel announce-port <port> dir /tmp sentinel monitor mymaster 127.0.0.1 6379 2 # sentinel auth-pass <master-name> <password> sentinel down-after-milliseconds mymaster 30000 sentinel parallel-syncs mymaster 1 sentinel failover-timeout mymaster 180000 # sentinel notification-script mymaster /var/redis/notify.sh # sentinel client-reconfig-script mymaster /var/redis/reconfig.sh sentinel deny-scripts-reconfig yes
其中，
dir：设置Sentinel的工作目录。
sentinel monitor mymaster 127.0.0.1 6379 2：其中2是quorum，即权重，代表至少需要两个Sentinel节点认为主节点主观下线，才可判定主节点为客观下线。一般建议将其设置为Sentinel节点的一半加1。不仅如此，quorum还与Sentinel节点的领导者选举有关。为了选出Sentinel的领导者，至少需要max(quorum,num(sentinels) / 2 + 1)个Sentinel节点参与选举。
sentinel down-after-milliseconds mymaster 30000：每个Sentinel节点都要通过定期发送ping命令来判断Redis节点和其余Sentinel节点是否可达。
如果在指定的时间内，没有收到主节点的有效回复，则判断其为主观下线。需要注意的是，该参数不仅用来判断主节点状态，同样也用来判断该主节点下面的从节点及其它Sentinel的状态。其默认值为30s。
sentinel parallel-syncs mymaster 1：在failover期间，允许多少个slave同时指向新的主节点。如果numslaves设置较大的话，虽然复制操作并不会阻塞主节点，但多个节点同时指向新的主节点，会增加主节点的网络和磁盘IO负载。
sentinel failover-timeout mymaster 180000：定义故障切换超时时间。默认180000，单位秒，即3min。需要注意的是，该时间不是总的故障切换的时间，而是适用于故障切换的多个场景。
# Specifies the failover timeout in milliseconds. It is used in many ways: # # - The time needed to re-start a failover after a previous failover was # already tried against the same master by a given Sentinel,is two # times the failover timeout. # # - The time needed for a slave replicating to a wrong master according # to a Sentinel current configuration,to be forced to replicate # with the right master,is exactly the failover timeout (counting since # the moment a Sentinel detected the misconfiguration). # # - The time needed to cancel a failover that is already in progress but # did not produced any configuration change (SLAVEOF NO ONE yet not # acknowledged by the promoted slave). # # - The maximum time a failover in progress waits for all the slaves to be # reconfigured as slaves of the new master. However even after this time # the slaves will be reconfigured by the Sentinels anyway,but not with # the exact parallel-syncs progression as specified.
第一种适用场景：如果Redis Sentinel对一个主节点故障切换失败，那么下次再对该主节点做故障切换的起始时间是failover-timeout的2倍。这点从Sentinel的日志就可体现出来（28234:X 08 Oct 16:04:04.385 # Next failover delay: I will not start a failover before Mon Oct 8 16:10:04 2018）
sentinel notification-script：定义通知脚本，当Sentinel出现WARNING级别的事件时，会调用该脚本，其会传入两个参数：事件类型，事件描述。
sentinel client-reconfig-script：当主节点发生切换时，会调用该参数定义的脚本，其会传入以下参数：<master-name> <role> <state> <from-ip> <from-port> <to-ip> <to-port>
关于脚本，其必须遵循一定的规则。
# SCRIPTS EXECUTION # # sentinel notification-script and sentinel reconfig-script are used in order # to configure scripts that are called to notify the system administrator # or to reconfigure clients after a failover. The scripts are executed # with the following rules for error handling: # # If script exits with "1" the execution is retried later (up to a maximum # number of times currently set to 10). # # If script exits with "2" (or an higher value) the script execution is # not retried. # # If script terminates because it receives a signal the behavior is the same # as exit code 1. # # A script has a maximum running time of 60 seconds. After this limit is # reached the script is terminated with a SIGKILL and the execution retried.
sentinel deny-scripts-reconfig：不允许使用SENTINEL SET设置notification-script和client-reconfig-script。
Sentinel的常见操作
- PING This command simply returns PONG.
- SENTINEL masters Show a list of monitored masters and their state.
- SENTINEL master <master name> Show the state and info of the specified master.
- SENTINEL slaves <master name> Show a list of slaves for this master,and their state.
- SENTINEL sentinels <master name> Show a list of sentinel instances for this master,1)">SENTINEL get-master-addr-by-name <master name> Return the ip and port number of the master with that name. If a failover is in progress or terminated successfully for this master it returns the address and port of the promoted slave.
- SENTINEL reset <pattern> This command will reset all the masters with matching name. The pattern argument is a glob-style pattern. The reset process clears any previous state in a master (including a failover in progress),and removes every slave and sentinel already discovered and associated with the master.
- SENTINEL failover <master name> Force a failover as if the master was not reachable,and without asking for agreement to other Sentinels (however a new version of the configuration will be published so that the other Sentinels will update their configurations).
- SENTINEL ckquorum <master name> Check if the current Sentinel configuration is able to reach the quorum needed to failover a master,and the majority needed to authorize the failover. This command should be used in monitoring systems to check if a Sentinel deployment is ok.
- SENTINEL flushconfig Force Sentinel to rewrite its configuration on disk,including the current Sentinel state. Normally Sentinel rewrites the configuration every time something changes in its state (in the context of the subset of the state which is persisted on disk across restart). However sometimes it is possible that the configuration file is lost because of operation errors,disk failures,package upgrade scripts or configuration managers. In those cases a way to to force Sentinel to rewrite the configuration file is handy. This command works even if the previous configuration file is completely missing.
- SENTINEL MONITOR <name> <ip> <port> <quorum> This command tells the Sentinel to start monitoring a new master with the specified name,ip,port,and quorum. It is identical to the sentinel monitor configuration directive in sentinel.conf configuration file
- SENTINEL REMOVE <name> is used in order to remove the specified master: the master will no longer be monitored,and will totally be removed from the internal state of the Sentinel,so it will no longer listed by SENTINEL masters and so forth.
- SENTINEL SET <name> <option> <value> The SET command is very similar to the CONFIG SET command of Redis,and is used in order to change configuration parameters of a specific master. Multiple option / value pairs can be specified (or none at all). All the configuration parameters that can be configured via sentinel.conf are also configurable using the SET command.
sentinel masters
输出被监控的主节点的状态信息
26379> sentinel masters 1) 1) "name" 2) mymaster3) ip4) 127.0.0.15) port6) 63797) runid8) 6ab2be5db3a37c10f2473c8fb9daed147a32df3e9) flags" 10) master11) link-pending-commands12) 013) link-refcount14) 115) last-ping-sent16) 17) last-ok-ping-reply18) 63919) last-ping-reply20) 21) down-after-milliseconds22) 3000023) info-refresh24) 207525) role-reported26) 27) role-reported-time28) 75968229) config-epoch30) 31) num-slaves32) 233) num-other-sentinels34) 35) quorum36) 37) failover-timeout38) 18000039) parallel-syncs40) "
也可单独查看某个主节点的状态
sentinel master mymaster
sentinel slaves mymaster
查看某个主节点slave的状态
sentinel slaves mymaster 127.0.0.1:63806380983b87fd070c7f052b26f5135bbb30fdeb170a54slave1786160489019master-link-down-timemaster-link-statusokmaster-hostmaster-portslave-priority100slave-repl-offset70375" 2) 127.0.0.1:63816381b88059cce9104dd4e0366afd6ad07a163dae8b15291871040"
sentinel sentinels mymaster
查看其它Sentinel的状态
sentinel sentinels mymaster 738ccbddaa0d4379d89a147613d9aecfec765bcb26381sentinel475last-hello-message79voted-leader?voted-leader-epoch7251bb129ca373ad0d8c7baf3b6577ae2593079f26380985"
sentinel get-master-addr-by-name <master name>
返回指定<master name>主节点的IP地址和端口。如果在进行故障切换，则显示的是新主的信息。
127.0.0.1:26379> sentinel get-master-addr-by-name mymaster
1) "127.0.0.1"
2) "6379"
sentinel reset <pattern>
对符合<pattern>（通配符风格）主节点的配置进行重置。
如果某个slave宕机了，其依然处于sentinel的管理中，所以，在其恢复正常后，其依然会加入到之前的复制环境中，即使配置文件中没有指定slaveof选项。不仅如此，如果主节点宕机了，在其重启后，其默认会作为从节点接入到之前的复制环境中。
但很多时候，我们可能就是想移除old master，slave，这个时候，sentinel reset就派上用场了。其会基于当前主节点的状态，重置其配置（they'll refresh the list of slaves within the next 10 seconds,only adding the ones listed as correctly replicating from the current master INFO output）。关键的是，对于非正常状态的slave，会从当前的配置中剔除。这样，被剔除节点在恢复正常后（注意此时的配置文件，需剔除slaveof的配置），也不会自动加入到之前的复制环境中。
需要注意的是，该命令仅对当前sentinel节点有效，如果要剔除某个节点，需要在所有的sentinel节点上执行reset操作。
sentinel failover <master name>
对指定 <master name> 主节点进行强制故障切换。相对于常规的故障切换，其无需进行Sentinel节点的领导者选举。直接由当前Sentinel节点进行后续的故障切换。
sentinel ckquorum <master name>
检测当前可达的Sentinel节点总数是否达到<quorum>的个数
127.0.0.1:26379> sentinel ckquorum mymaster
OK 3 usable Sentinels. Quorum and failover authorization can be reached
sentinel flushconfig
将Sentinel节点的配置信息强制刷到磁盘上，这个命令Sentinel节点自身用得比较多，对于开发和运维人员只有当外部原因（例如磁盘损坏）造成配置文件损坏或者丢失时，才会用上。
sentinel remove <master name>
取消当前Sentinel节点对于指定<master name>主节点的监控。
[root@slowtech redis-4.0.11]# grep -Ev ^#|^$" sentinel_26379.conf port 26379 dir /tmp sentinel myid 2467530fa249dbbc435c50fbb0dc2a4e766146f8 sentinel deny-scripts-reconfig yes sentinel monitor mymaster 2 sentinel config-epoch mymaster 12 sentinel leader-epoch mymaster 0 sentinel known-slave mymaster 63806379 sentinel known-sentinel mymaster 26381 738ccbddaa0d4379d89a147613d9aecfec765bcb sentinel known-sentinel mymaster 26380 7251bb129ca373ad0d8c7baf3b6577ae2593079f sentinel current-epoch [root@slowtech redis-11]# redis-cli -p 26379 sentinel remove mymaster OK quit [root@slowtech redis-reconfig yes sentinel current-epoch 12
sentinel set <name> <option> <value>
参数 用法
quorum sentinel set mymaster quorum 3
down-after-milliseconds sentinel set mymaster down-after-milliseconds 30000
failover-timeout sentinel set mymaster failover-timeout 18000
parallel-syncs sentinel set mymaster parallel-syncs 3
notification-script sentinel set mymaster notification-script /tmp/a.sh
client-reconfig-script sentinel set mymaster client-reconfig-script /tmp/b.sh
auth-pass sentinel set mymaster auth-pass masterpassword
需要注意的是：
1. sentinel set命令只对当前Sentinel节点有效。
2. sentinel set命令如果执行成功会立即刷新配置文件，这点和Redis普通数据节点不同，后者修改完配置后，需要执行config rewrite刷新到配置文件。
3. 建议所有Sentinel节点的配置尽可能一致。
4. Sentinel不支持config命令。如何要查看参数的设置，可痛过SENTINEL MASTER命令查看。
参考：
1. 《Redis开发与运维》
2. 《Redis设计与实现》
3. 《Redis 4.X Cookbook》
4. 官方文档
原文地址：https://www.cnblogs.com/ivictor
版权声明：本文内容由互联网用户自发贡献，该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容， 请发送邮件至 dio@foxmail.com 举报，一经查实，本站将立刻删除。