部署 codis 集群简介

集群需要依赖于 zookeeper 或者 etcd 服务。这里使用的是 zookeeper 。

zookeeper 最好选择使用集群模式部署，提高可用性。

codis-dashboard 通过操作 zookeeper 保存 dashboard、proxy、slot 等相关信息。

通过 codis-dashboard 可以添加 codis-proxy、codis-server、分组、sentinel、迁移 slot 等。

codis-fe 可以通过 zookeeper，获取对应的信息，来展示，并获取 dashboard 地址，来进行控制。

Codis 下载地址

wget -c https://github.com/CodisLabs/codis/releases/download/3.2.2/codis3.2.2-go1.9.2-linux.tar.gz
tar zxvf codis3.2.2-go1.9.2-linux.tar.gz

cd codis3.2.2-go1.9.2-linux
mkdir etc

配置 codis-dashboard

可以使用 codis-dashboard 生成默认配置文件

./codis-dashboard --default-config  | tee etc/dashboard.toml

修改对应的 zookeeper 地址，为了安全性可以修改 product_name 和 product_auth，整个集群的 product 相关信息要保持一致。

##################################################
#                                                #
#                  Codis-Dashboard               #
#                                                #
##################################################

# Set Coordinator, only accept "zookeeper" & "etcd" & "filesystem".
# for zookeeper/etcd, coorinator_auth accept "user:password"
# Quick Start
#coordinator_name = "filesystem"
#coordinator_addr = "/tmp/codis"
# 这里修改zookeeper的地址
coordinator_name = "zookeeper"
coordinator_addr = "192.168.122.101:2181,192.168.122.102:2181,192.168.122.103:2181"
#coordinator_auth = ""

# Set Codis Product Name/Auth.
product_name = "codis3"
product_auth = "h6oewuoaudPgCJRzYg0Uo"

# Set bind address for admin(rpc), tcp only.
admin_addr = "0.0.0.0:18080"

# Set arguments for data migration (only accept 'sync' & 'semi-async').
migration_method = "semi-async"
migration_parallel_slots = 100
migration_async_maxbulks = 200
migration_async_maxbytes = "32mb"
migration_async_numkeys = 500
migration_timeout = "30s"

# Set configs for redis sentinel.
sentinel_client_timeout = "10s"
sentinel_quorum = 2
sentinel_parallel_syncs = 1
sentinel_down_after = "30s"
sentinel_failover_timeout = "5m"
sentinel_notification_script = ""
sentinel_client_reconfig_script = ""

启动命令

nohup ./codis-dashboard --ncpu=2 --config=./etc/dashboard.toml --log=/data/codis/logs/dashboard.log --log-level=WARN &

--ncpu 可以指定使用几个 CPU
--config 指定配置文件的路径
--log 指定日志文件路径
--log-devel 指定日志级别

配置 codis-proxy

使用 codis-proxy 生成默认配置文件

./codis-proxy  --default-config   | tee etc/proxy.toml

修改 product_name 和 product_auth，整个集群的 product 相关信息要保持一致。session_auth 是用于连接的验证，可以跟集群的验证不一样。如果短连接比较多可以调整一下 proxy_max_clients 参数。

##################################################
#                                                #
#                  Codis-Proxy                   #
#                                                #
##################################################

# Set Codis Product Name/Auth.
product_name = "codis3"
product_auth = "h6oewuoaudPgCJRzYg0Uo"

# Set auth for client session
#   1. product_auth is used for auth validation among codis-dashboard,
#      codis-proxy and codis-server.
#   2. session_auth is different from product_auth, it requires clients
#      to issue AUTH <PASSWORD> before processing any other commands.
session_auth = "L5SqVOieIOk7IyfYscCJcCdQBtHlK5Uq"

# Set bind address for admin(rpc), tcp only.
admin_addr = "0.0.0.0:11080"

# Set bind address for proxy, proto_type can be "tcp", "tcp4", "tcp6", "unix" or "unixpacket".
proto_type = "tcp4"
proxy_addr = "0.0.0.0:19000"

# Set jodis address & session timeout
#   1. jodis_name is short for jodis_coordinator_name, only accept "zookeeper" & "etcd".
#   2. jodis_addr is short for jodis_coordinator_addr
#   3. jodis_auth is short for jodis_coordinator_auth, for zookeeper/etcd, "user:password" is accepted.
#   4. proxy will be registered as node:
#        if jodis_compatible = true (not suggested):
#          /zk/codis/db_{PRODUCT_NAME}/proxy-{HASHID} (compatible with Codis2.0)
#        or else
#          /jodis/{PRODUCT_NAME}/proxy-{HASHID}
jodis_name = ""
jodis_addr = ""
jodis_auth = ""
jodis_timeout = "20s"
jodis_compatible = false

# Set datacenter of proxy.
proxy_datacenter = ""

# Set max number of alive sessions.
proxy_max_clients = 10000

# Set max offheap memory size. (0 to disable)
proxy_max_offheap_size = "1024mb"

# Set heap placeholder to reduce GC frequency.
proxy_heap_placeholder = "256mb"

# Proxy will ping backend redis (and clear 'MASTERDOWN' state) in a predefined interval. (0 to disable)
backend_ping_period = "5s"

# Set backend recv buffer size & timeout.
backend_recv_bufsize = "128kb"
backend_recv_timeout = "30s"

# Set backend send buffer & timeout.
backend_send_bufsize = "128kb"
backend_send_timeout = "30s"

# Set backend pipeline buffer size.
backend_max_pipeline = 20480

# Set backend never read replica groups, default is false
backend_primary_only = false

# Set backend parallel connections per server
backend_primary_parallel = 1
backend_replica_parallel = 1

# Set backend tcp keepalive period. (0 to disable)
backend_keepalive_period = "75s"

# Set number of databases of backend.
backend_number_databases = 16

# If there is no request from client for a long time, the connection will be closed. (0 to disable)
# Set session recv buffer size & timeout.
session_recv_bufsize = "128kb"
session_recv_timeout = "30m"

# Set session send buffer size & timeout.
session_send_bufsize = "64kb"
session_send_timeout = "30s"

# Make sure this is higher than the max number of requests for each pipeline request, or your client may be blocked.
# Set session pipeline buffer size.
session_max_pipeline = 10000

# Set session tcp keepalive period. (0 to disable)
session_keepalive_period = "75s"

# Set session to be sensitive to failures. Default is false, instead of closing socket, proxy will send an error response to client.
session_break_on_failure = false

# Set metrics server (such as http://localhost:28000), proxy will report json formatted metrics to specified server in a predefined period.
metrics_report_server = ""
metrics_report_period = "1s"

# Set influxdb server (such as http://localhost:8086), proxy will report metrics to influxdb.
metrics_report_influxdb_server = ""
metrics_report_influxdb_period = "1s"
metrics_report_influxdb_username = ""
metrics_report_influxdb_password = ""
metrics_report_influxdb_database = ""

# Set statsd server (such as localhost:8125), proxy will report metrics to statsd.
metrics_report_statsd_server = ""
metrics_report_statsd_period = "1s"
metrics_report_statsd_prefix = ""

启动命令

nohup ./codis-proxy --ncpu=2 --config=./etc/proxy.toml --log=/data/codis/logs/proxy.log --log-level=WARN &

--ncpu 可以指定使用几个 CPU
--config 指定配置文件的路径
--log 指定日志文件路径
--log-devel 指定日志级别

配置 codis-server

codis-server 跟 redis-server 配置是一样的。设置的密码要跟 product_auth 这个参数一样，集群才能验证通过。

例如配置一个 7001.conf 的配置文件，如果多个节点，复制复制一份，替换文本中 7001 关键字即可。

daemonize yes
pidfile "/data/codis/redis_7001.pid"
port 7001
timeout 60
loglevel warning
logfile "/data/codis/logs/codis-server-7001.log"
databases 16
rdbcompression yes
dbfilename "dump-7001.rdb"
dir "/data/codis/data"
maxmemory 5gb
maxmemory-policy volatile-lru
appendonly yes
appendfilename "appendonly7001.aof"
appendfsync everysec
no-appendfsync-on-rewrite yes
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
slowlog-log-slower-than 10000
slowlog-max-len 1024
hz 50
aof-rewrite-incremental-fsync yes
requirepass "h6oewuoaudPgCJRzYg0Uo"
# Generated by CONFIG REWRITE
masterauth "h6oewuoaudPgCJRzYg0Uo"

启动命令

./codis-server etc/7001.conf
...

配置 codis-fe

nohup ./codis-fe --ncpu=2 --log=/data/codis/logs/fe.log --log-level=WARN --zookeeper=192.168.122.101:2181,192.168.122.102:2181,192.168.122.103:2181  --listen=0.0.0.0:8080 &

--ncpu 指定CPU个数
--log 指定日志文件
--log-level 日志级别
--zookeeper 指定zookeeper地址
--listen 监听地址:端口

配置 sentinel

cat > etc/sentinel.conf
port 26379
protected-mode no
daemonize yes

启动命令

/redis-sentinel etc/sentinel.conf

Web 界面

在 codis-fe 启动的节点，访问 web 页面。

通过 New Proxy 添加 codis-proxy 节点，默认端口是 11080

通过 New Group 添加分组 ，按照数字 1 2 3 …来即可。

通过 Add Server 添加节点到对应的分组，例如 192.168.122.101:7001 to 1

通过 Migrate Slots 分配 slot 到对应的分组，例如：0 ~ 40 to 1。

通过 Add Sentinel 添加 sentinel，例如：192.168.122.101:26379，添加后，主节点会有 HA 的标记