redis-sentinel 소개 및 설정(설치 추가)

레디스-센티넬 소개 및 설정.

redis-sentinel이란?

Redis 인스턴스 관리를 돕기 위해  디자인된 시스템으로, 레디스의 fail-over를 돕는 도구라고 보시면 됩니다.

보통 홀수로 구성하여 과반의 센티넬이 마스터 노드가 죽었다고 판단하면 후로 리플리케이션 중 하나를 master로 올리는 형태를 취합니다.

(2020.03.20 추가 : 업무에서 사용하는 관계로 블로그를 찾아보는데, 설치는 진행되지 않았더군요. Ubuntu를 기준으로 설치 스크립트를 추가합니다(매우간단..)

 apt install redis-sentinel

레디스가 4.0.6으로 업그레이드 되며, 관련 설정 또한 주석처리되어서 안내되어 굳이 (물론 필요한 경우는 열어야 겠지만)

sentinel 관련 레퍼런스를 안열어도 이해하기 쉽도록 변경되었습니다.

아래는 센티넬 설정파일이며, 관련해 주석을 달아 어떤 내용인지 적어두었습니다.(센티넬 설정파일의 경로는 /etc/redis/sentinel.conf 입니다)

# Example sentinel.conf
 
# *** IMPORTANT ***
#
# By default Sentinel will not be reachable from interfaces different than
# localhost, either use the 'bind' directive to bind to a list of network
# interfaces, or disable protected mode with "protected-mode no" by
# adding it to this configuration file.
#
# Before doing that MAKE SURE the instance is protected from the outside
# world via firewalling or other means.
#
# For example you may use one of the following:
#
# bind 127.0.0.1 192.168.1.1
#
# protected-mode no
bind 123.456.789.012 127.0.0.1
# 바인드 : 자신이 통신을 받을 IP 주소 명시. ifconfig를 참조하여 IP 정의하면 됨.
# port <sentinel-port>
# The port that this sentinel instance will run on
port 26379
# 포트번호
# sentinel announce-ip <ip>
# sentinel announce-port <port>
#
# The above two configuration directives are useful in environments where,
# because of NAT, Sentinel is reachable from outside via a non-local address.
#
# When announce-ip is provided, the Sentinel will claim the specified IP address
# in HELLO messages used to gossip its presence, instead of auto-detecting the
# local address as it usually does.
#
# Similarly when announce-port is provided and is valid and non-zero, Sentinel
# will announce the specified TCP port.
#
# The two options don't need to be used together, if only announce-ip is
# provided, the Sentinel will announce the specified IP and the server port
# as specified by the "port" option. If only announce-port is provided, the
# Sentinel will announce the auto-detected local IP and the specified port.
#
# Example:
#
# sentinel announce-ip 1.2.3.4
 
# dir <working-directory>
# Every long running process should have a well-defined working directory.
# For Redis Sentinel to chdir to /tmp at startup is the simplest thing
# for the process to don't interfere with administrative tasks such as
# unmounting filesystems.
dir "/data/redis/tmp"
# 작업 디렉토리로, 디폴트는 /tmp이나, 스토리지 용량이 넓은 곳으로 이관
# sentinel monitor <master-name> <ip> <redis-port> <quorum>
#
# Tells Sentinel to monitor this master, and to consider it in O_DOWN
# (Objectively Down) state only if at least <quorum> sentinels agree.
#
# Note that whatever is the ODOWN quorum, a Sentinel will require to
# be elected by the majority of the known Sentinels in order to
# start a failover, so no failover can be performed in minority.
#
# Slaves are auto-discovered, so you don't need to specify slaves in
# any way. Sentinel itself will rewrite this configuration file adding
# the slaves using additional configuration options.
# Also note that the configuration file is rewritten when a
# slave is promoted to master.
#
# Note: master name should not include special characters or spaces.
# The valid charset is A-z 0-9 and the three characters ".-_".
sentinel myid 37a7b04d44a19db6b1dd5a85a2d148e193b567ad
 
sentinel monitor mymaster 192.168.25.104 6379 2
# 모니터할 IP와 odown 의 투표 수 기준
# sdown : 주관적인 다운, sentinel 하나만 down이라고 판단. 
# odown : 명백한 다운, 제일 마지막 숫자 이상의 센티넬이 down이라고 판단하면 결정됨.
# sentinel auth-pass <master-name> <password>
#
# Set the password to use to authenticate with the master and slaves.
# Useful if there is a password set in the Redis instances to monitor.
#
# Note that the master password is also used for slaves, so it is not
# possible to set a different password in masters and slaves instances
# if you want to be able to monitor these instances with Sentinel.
#
# However you can have Redis instances without the authentication enabled
# mixed with Redis instances requiring the authentication (as long as the
# password set is the same for all the instances requiring the password) as
# the AUTH command will have no effect in Redis instances with authentication
# switched off.
#
# Example:
#
# sentinel auth-pass mymaster MySUPER--secret-0123passw0rd
 
 
 
sentinel down-after-milliseconds mymaster 10000
# 마스터 다운 후 명백히 sdown으로 판단하는 시간으로, 디폴트는 30초이나 10초로 설정
# sentinel down-after-milliseconds <master-name> <milliseconds>
#
# Number of milliseconds the master (or any attached slave or sentinel) should
# be unreachable (as in, not acceptable reply to PING, continuously, for the
# specified period) in order to consider it in S_DOWN state (Subjectively
# Down).
#
# Default is 30 seconds.
sentinel failover-timeout mymaster 15000
 
 
# sentinel parallel-syncs <master-name> <numslaves>
#
# How many slaves we can reconfigure to point to the new slave simultaneously
# during the failover. Use a low number if you use the slaves to serve query
# to avoid that all the slaves will be unreachable at about the same
# time while performing the synchronization with the master.
sentinel auth-pass mymaster mypassword!@@#!4
# redis접속에 auth 정보가 필요할 경우 마스터의 auth정보를 명시
 
# sentinel failover-timeout <master-name> <milliseconds>
#
# Specifies the failover timeout in milliseconds. It is used in many ways:
#
# - The time needed to re-start a failover after a previous failover was
#   already tried against the same master by a given Sentinel, is two
#   times the failover timeout.
#
# - The time needed for a slave replicating to a wrong master according
#   to a Sentinel current configuration, to be forced to replicate
#   with the right master, is exactly the failover timeout (counting since
#   the moment a Sentinel detected the misconfiguration).
#
# - The time needed to cancel a failover that is already in progress but
#   did not produced any configuration change (SLAVEOF NO ONE yet not
#   acknowledged by the promoted slave).
#
# - The maximum time a failover in progress waits for all the slaves to be
#   reconfigured as slaves of the new master. However even after this time
#   the slaves will be reconfigured by the Sentinels anyway, but not with
#   the exact parallel-syncs progression as specified.
#
# Default is 3 minutes.
sentinel config-epoch mymaster 2
sentinel leader-epoch mymaster 2
# SCRIPTS EXECUTION
#
# sentinel notification-script and sentinel reconfig-script are used in order
# to configure scripts that are called to notify the system administrator
# or to reconfigure clients after a failover. The scripts are executed
# with the following rules for error handling:
#
# If script exits with "1" the execution is retried later (up to a maximum
# number of times currently set to 10).
#
# If script exits with "2" (or an higher value) the script execution is
# not retried.
#
# If script terminates because it receives a signal the behavior is the same
# as exit code 1.
#
# A script has a maximum running time of 60 seconds. After this limit is
# reached the script is terminated with a SIGKILL and the execution retried.
 
# NOTIFICATION SCRIPT
#
# sentinel notification-script <master-name> <script-path>
#
# Call the specified notification script for any sentinel event that is
# generated in the WARNING level (for instance -sdown, -odown, and so forth).
# This script should notify the system administrator via email, SMS, or any
# other messaging system, that there is something wrong with the monitored
# Redis systems.
#
# The script is called with just two arguments: the first is the event type
# and the second the event description.
#
# The script must exist and be executable in order for sentinel to start if
# this option is provided.
#
# Example:
#
# sentinel notification-script mymaster /var/redis/notify.sh
 
# CLIENTS RECONFIGURATION SCRIPT
#
# sentinel client-reconfig-script <master-name> <script-path>
#
# When the master changed because of a failover a script can be called in
# order to perform application-specific tasks to notify the clients that the
# configuration has changed and the master is at a different address.
#
# The following arguments are passed to the script:
#
# <master-name> <role> <state> <from-ip> <from-port> <to-ip> <to-port>
#
# <state> is currently always "failover"
# <role> is either "leader" or "observer"
#
# The arguments from-ip, from-port, to-ip, to-port are used to communicate
# the old address of the master and the new address of the elected slave
# (now a master).
#
# This script should be resistant to multiple invocations.
#
# Example:
#
# sentinel client-reconfig-script mymaster /var/redis/reconfig.sh
 
daemonize yes
pidfile "/var/run/sentinel.pid"
logfile "/data/redis/log/sentinel.log"
# 데몬화를 위한 옵션
 

Sentinel 부팅시 실행을 위한 옵션(redis 실행 파일을 복사해서 사용

아래는 센티넬을 손쉽게 사용하기 위해 시작시 수행되도록 /etc/init.d에 설정하는 쉘 파일입니다. 아래와 같이 설정하여 더욱 편하게 구동 및 재구동, 셧다운을 진행하실 수 있습니다.

#!/bin/sh
#Configurations injected by install_server below....
 
EXEC=/redis-4.0.6/src/redis-sentinel
CLIEXEC=/redis-4.0.6/src/redis-cli
PIDFILE=/var/run/sentinel.pid
CONF="/etc/redis/sentinel.conf"
REDISPORT="26379"
###############
# SysV Init Information
# chkconfig: - 58 74
# description: redis_6379 is the redis daemon.
### BEGIN INIT INFO
# Provides: redis_6379
# Required-Start: $network $local_fs $remote_fs
# Required-Stop: $network $local_fs $remote_fs
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Should-Start: $syslog $named
# Should-Stop: $syslog $named
# Short-Description: start and stop redis_6379
# Description: Redis daemon
### END INIT INFO
 
 
case "$1" in
    start)
        if [ -f $PIDFILE ]
        then
            echo "$PIDFILE exists, process is already running or crashed"
        else
            echo "Starting Redis Sentinel server..."
            $EXEC $CONF
        fi
        ;;
    stop)
        if [ ! -f $PIDFILE ]
        then
            echo "$PIDFILE does not exist, process is not running"
        else
            PID=$(cat $PIDFILE)
            echo "Stopping ..."
            $CLIEXEC -p $REDISPORT shutdown
            while [ -x /proc/${PID} ]
            do
                echo "Waiting for Redis Sentinel to shutdown ..."
                sleep 1
            done
            echo "Redis Sentinel stopped"
        fi
        ;;
    status)
        PID=$(cat $PIDFILE)
        if [ ! -x /proc/${PID} ]
        then
            echo 'Redis Sentinel is not running'
        else
            echo "Redis Sentinel is running ($PID)"
        fi
        ;;
    restart)
        $0 stop
        $0 start
        ;;
    *)
        echo "Please use start, stop, restart or status as first argument"
        ;;
esac
 

실행방법

$ redis-sentinel /path/to/sentinel.conf

OR

$ redis-server /path/to/sentinel.conf --sentinel

Automatic FailOver 테스트

테스트 구성

REDIS

Port : 6379

REDIS-sentinel

Port : 26379

MonitoringREDIS

Port : 6379

REDIS-sentinel

Port : 26379

MonitoringREDIS

Port : 6379

REDIS-sentinel

Port : 26379

MonitoringreplicationingSLAVE1MASTERSLAVE2

replicationing

시나리오

sentinel 실행

master shutdown

failover 확인

sentinel 실행

/etc/init.d/sentinel_26379 start

master shutdown

 /REDIS/PATH/src/redis-cli 

127.0.0.1:6379> shutdown

not connected> exit

failover 확인

Sentinel 구성시 주의사항

1. bind 설정 필수 필요

다수 센티널 구성시, bind설정을 통해 서로 통신이 가능하게끔 방화벽 오픈 등을 설정해야 합니다..

2. master-slave 간 비밀번호 동기화

master-slave간 auth 키(비밀번호)가 같아야 HA작동시 무리없이 리플리케이션을 옮겨서 붙습니다.

3. master node에도 auth변수 저장

마스터가 죽었다가 다시 살아날 시 자동으로 리플리케이션으로 설정되는데, 이 때 리플리케이션 정상 연결을 위하여 리플리케이션 auth(비밀번호) 변수 지정이 필요합니다.