Centos7，部署etcd集群，基于二进制包，https安全通讯

由于etcd集群https通讯，所以需要自建CA数字证书，学习使用https部署etcd集群前，可以先完成一下，基于http通信的etcd集群：

关于CA原理以及工作可以阅读，以下两篇文章：
CA工作原理
对称加密与非对称加密

关于如何自建CA数字证书可以参考: 手动生成证书，文章介绍了，3种创建证书的方法，本文选择 openssl 来创建证书。

主机信息

etcd-01	192.168.44.186
etcd-02	192.168.44.187
etcd-03	192.168.44.188

环境配置：

1. 设置主机名

 # 分别在每台执行hostnamectl set-hostname etcd-01hostnamectl set-hostname etcd-02hostnamectl set-hostname etcd-03

2. 配置本地解析

# 每台都执行
cat >> /etc/hosts << EOF
192.168.44.186  etcd-01
192.168.44.187  etcd-02
192.168.44.188  etcd-03
EOF

3. 关闭防火墙，selinux

 # 每台都执行systemctl disable firewalld --nowsetenforce 0sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config

4. 配置时间同步

 # 每台都执行yum install chrony -y && systemctl enable --now chronyd

下载二进制包，并创建启动目录

# 每台都执行
cd /opt
wget https://github.com/etcd-io/etcd/releases/download/v3.5.13/etcd-v3.5.13-linux-amd64.tar.gz
tar -zxvf etcd-v3.5.13-linux-amd64.tar.gz
# 创建etcd工作目录： bin存放二进制目录，cfg存放配置文件目录，ssl存放证书目录
mkdir -p  /usr/local/etcd/{bin,cfg,ssl}
# 将二进制文件送到  /usr/local/etcd/bin目录
cp etcd-v3.5.13-linux-amd64/etcd* /usr/local/etcd/bin/
# 配置环境变量
echo 'export ETCD_HOME=/usr/local/etcd' >> /etc/profile
echo 'export PATH=$PATH:$ETCD_HOME/bin' >> /etc/profile
source /etc/profile
# 现在可以查看etcd版本
etcd --version

生成证书
可以选择在任意台节点上生成证书，然后copy到其他节点。

# 在节点etcd-01上执行cd /usr/local/etcd/ssl/
# 生成CA私钥openssl genrsa -out ca.key 2048
# 使用CA私钥生成CA根证书， -days 设置过期时间openssl req -x509 -new -nodes -key ca.key -subj "/CN=etcd" -days 10000 -out ca.crt 
# 生成服务端私钥openssl genrsa -out server.key 2048
# 创建一个用于生成证书签名请求（CSR）的配置文件，etcd-csr.conf，内容如下：
# 需要注意 [ alt_names ] 块，为集群的成员，需要把所有的集群成员IP都写进去，也可以多写入，为以后做扩展

[ req ]
default_bits = 2048
prompt = no
default_md = sha256
req_extensions = req_ext
distinguished_name = dn[ dn ]
C = cn
ST = fujian
L = xiamen
O = etcd
OU = etcd
CN = etcd[ req_ext ]
subjectAltName = @alt_names[ alt_names ]
IP.1 = 192.168.44.186
IP.2 = 192.168.44.187
IP.3 = 192.168.44.188[ v3_ext ]
authorityKeyIdentifier=keyid,issuer:always
basicConstraints=CA:FALSE
keyUsage=keyEncipherment,dataEncipherment
extendedKeyUsage=serverAuth,clientAuth
subjectAltName=@alt_names

# 基于etcd-csr.conf证书请求配置文件，生成证书签名请求：openssl req -new -key server.key -out server.csr -config etcd-csr.conf
# 基于 ca.key、ca.crt 和 server.csr 等三个文件生成服务端证书：openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key  -CAcreateserial -out server.crt -days 10000 -extensions v3_ext -extfile etcd-csr.conf -sha256# 查看所有的文件

[root@localhost ssl]# ll
总用量 28
-rw-r--r--. 1 root root 1082 4月  12 14:26 ca.crt
-rw-r--r--. 1 root root 1675 4月  12 14:24 ca.key
-rw-r--r--. 1 root root   17 4月  12 14:36 ca.srl
-rw-r--r--. 1 root root  486 4月  12 14:35 etcd-csr.conf
-rw-r--r--. 1 root root 1285 4月  12 14:36 server.crt
-rw-r--r--. 1 root root 1050 4月  12 14:35 server.csr
-rw-r--r--. 1 root root 1679 4月  12 14:26 server.key

# 查看证书签名请求：openssl req  -noout -text -in ./server.csr
# 查看证书：openssl x509  -noout -text -in ./server.crt
# 将所有证书文件分发到 etcd-02 , etcd-03 节点scp * etcd-02:/usr/local/etcd/ssl/scp * etcd-03:/usr/local/etcd/ssl/

启动集群

1. 创建etcd配置文件

如果有部署http协议的etcd集群的经验，观察配置文件就可以看出，就是配置了变量，将之前运行时的参数写入到了配置文件中， 协议由http改成了https。

# etcd-01
cat > /usr/local/etcd/cfg/etcd.conf << EOF
#[Member]
ETCD_NAME="etcd-01"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.44.186:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.44.186:2379"#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.44.186:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.44.186:2379"
ETCD_INITIAL_CLUSTER="etcd-01=https://192.168.44.186:2380,etcd-02=https://192.168.44.187:2380,etcd-03=https://192.168.44.188:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
EOF

# etcd-02
cat > /usr/local/etcd/cfg/etcd.conf << EOF
#[Member]
ETCD_NAME="etcd-02"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.44.187:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.44.187:2379"#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.44.187:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.44.187:2379"
ETCD_INITIAL_CLUSTER="etcd-01=https://192.168.44.186:2380,etcd-02=https://192.168.44.187:2380,etcd-03=https://192.168.44.188:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
EOF

# etcd-03
cat > /usr/local/etcd/cfg/etcd.conf << EOF
#[Member]
ETCD_NAME="etcd-03"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.44.188:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.44.188:2379"#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.44.188:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.44.188:2379"
ETCD_INITIAL_CLUSTER="etcd-01=https://192.168.44.186:2380,etcd-02=https://192.168.44.187:2380,etcd-03=https://192.168.44.188:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
EOF

参数详情讲解：

ETCD_NAME	      				etcd的节点名字,自定义名字,etcd节点的名字不能相同
ETCD_DATA_DIR    				etcd的数据存储目录
ETCD_LISTEN_CLIENT_URLS			用于当前节点与客户端交互的URL地址，每个节点同样可以向客户端提供多个URL地址，多个地址使用道号分隔节课,端口一般保持默认2379即可
ETCD_LISTEN_PEER_URLS	   		用于集群内各个节点之间通信的URL地址，每个节点可以监听多个URL地址,集群内部将通过这些URL地址进行数据交互,例如，Leader节点的选举、Message消息传输或是快照传输等,端口一般保持默认2380即可
ETCD_ADVERTISE_CLIENT_URLS		建议使用的客户端通信url,该值用于etcd 代理或etcd成员与etcd节点通信,与listen-client-urls参数值保持一致即可
ETCD_INITIAL_ADVERTISE_PEER_URLS	建议用于集群内部节点之间交互的URL地址,节点间将以该值进行通信,与listen-peer-urls参数值保持一致即可
ETCD_INITIAL_CLUSTER				集群中所有的initial-advertise-peer-urls的合集,etcd启动的时候,会通过这个配置找到其他etcd节点的列表
ETCD_INITIAL_CLUSTER_TOKEN 			节点的token值,该值可自定义,设置该值后集群将生成唯一id,并为每个节点也生成唯一id,当使用相同配置文件再启动一个集群时,只要该token值不一样,etcd 集群就不会相互影响
ETCD_INITIAL_CLUSTER_STATE			初始化时集群的状态,可取值:new和existing,new代表新建的集群,existing 代表加入已经存在的集群

2. 创建etcd.service文件，将etcd服务加入到systemd系统进程

# 每台节点都执行
cat > /usr/lib/systemd/system/etcd.service << EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target[Service]
Type=notify
EnvironmentFile=/usr/local/etcd/cfg/etcd.conf
ExecStart=/usr/local/etcd/bin/etcd \
--cert-file=/usr/local/etcd/ssl/server.crt \
--key-file=/usr/local/etcd/ssl/server.key \
--peer-cert-file=/usr/local/etcd/ssl/server.crt \
--peer-key-file=/usr/local/etcd/ssl/server.key \
--trusted-ca-file=/usr/local/etcd/ssl/ca.crt \
--peer-trusted-ca-file=/usr/local/etcd/ssl/ca.crt \
--logger=zap
Restart=on-failure
LimitNOFILE=65536[Install]
WantedBy=multi-user.target
EOF
# 加载systemd 配置文件
systemctl daemon-reload

参数详情讲解：

EnvironmentFile		配置文件路径
ExecStart			二进制启动文件路径
--cert-file			客户端服务器TLS证书文件的路径
--key-file			客户端服务器TLS密钥文件的路径
--peer-cert-file	与集群其他成员通讯TLS证书文件的路径
--peer-key-file		与集群其他成员通讯TLS密钥文件的路径
--trusted-ca-file	客户端服务器TLS可信CA证书文件的路径
--peer-trusted-ca-file		与集群其他成员通讯TLS可信CA证书文件的路径
--logger=zap		zap结构化日志记录（目前只支持“zap”进行结构化日志记录。）

3. 启动

    # 同时启动etcdsystemctl start etcd# 自启systemctl enable etcd
   

查看etcd集群状态

etcdctl \--cacert="/usr/local/etcd/ssl/ca.crt"  \--cert="/usr/local/etcd/ssl/server.crt" \--key="/usr/local/etcd/ssl/server.key" \--endpoints="https://192.168.44.186:2379,https://192.168.44.187:2379,https://192.168.44.186:2379" \endpoint status  --write-out=table

+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|          ENDPOINT           |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://192.168.44.186:2379 | 21465eba80fe786a |  3.5.13 |   25 kB |     false |      false |         2 |          8 |                  8 |        |
| https://192.168.44.187:2379 | 5d144d32bb3de813 |  3.5.13 |   25 kB |     false |      false |         2 |          8 |                  8 |        |
| https://192.168.44.186:2379 | 21465eba80fe786a |  3.5.13 |   25 kB |     false |      false |         2 |          8 |                  8 |        |
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

查看etcd集群是否健康

etcdctl \--cacert="/usr/local/etcd/ssl/ca.crt"  \--cert="/usr/local/etcd/ssl/server.crt" \--key="/usr/local/etcd/ssl/server.key" \--endpoints="https://192.168.44.186:2379,https://192.168.44.187:2379,https://192.168.44.186:2379" \endpoint health  --write-out=table

+-----------------------------+--------+-------------+-------+
|          ENDPOINT           | HEALTH |    TOOK     | ERROR |
+-----------------------------+--------+-------------+-------+
| https://192.168.44.187:2379 |   true |  17.89323ms |       |
| https://192.168.44.186:2379 |   true |  20.69533ms |       |
| https://192.168.44.186:2379 |   true | 21.574465ms |       |
+-----------------------------+--------+-------------+-------+

列出节点成员

etcdctl \--cacert="/usr/local/etcd/ssl/ca.crt"  \--cert="/usr/local/etcd/ssl/server.crt" \--key="/usr/local/etcd/ssl/server.key" \--endpoints="https://192.168.44.186:2379,https://192.168.44.187:2379,https://192.168.44.186:2379" \member list  --write-out=table

+------------------+---------+---------+-----------------------------+-----------------------------+------------+
|        ID        | STATUS  |  NAME   |         PEER ADDRS          |        CLIENT ADDRS         | IS LEARNER |
+------------------+---------+---------+-----------------------------+-----------------------------+------------+
| 21465eba80fe786a | started | etcd-01 | https://192.168.44.186:2380 | https://192.168.44.186:2379 |      false |
| 51f535e47d80fd93 | started | etcd-03 | https://192.168.44.188:2380 | https://192.168.44.188:2379 |      false |
| 5d144d32bb3de813 | started | etcd-02 | https://192.168.44.187:2380 | https://192.168.44.187:2379 |      false |
+------------------+---------+---------+-----------------------------+-----------------------------+------------+

为etcdctl设置别名
在使用etcdctl时，默认要加入证书参数，集群参数很不方便，所以可以把那串固定的指令设置一个别名
例如：
临时设置别名：

取消别名： unalias etcdctl

alias etcdctl='etcdctl \--cacert="/usr/local/etcd/ssl/ca.crt"  \--cert="/usr/local/etcd/ssl/server.crt" \--key="/usr/local/etcd/ssl/server.key" \--endpoints="https://192.168.44.186:2379,https://192.168.44.187:2379,https://192.168.44.186:2379"'

永久设置别名
将指令添加到环境变量文件即可。
例如：

# 添加到用户环境变量
cat >> ~/.bashrc << EOF
alias etcdctl='etcdctl \--cacert="/usr/local/etcd/ssl/ca.crt"  \--cert="/usr/local/etcd/ssl/server.crt" \--key="/usr/local/etcd/ssl/server.key" \--endpoints="https://192.168.44.186:2379,https://192.168.44.187:2379,https://192.168.44.186:2379"'
EOF
source ~/.bashrc# 或者添加上全局变量
cat >> /etc/profile << EOF
alias etcdctl='etcdctl \--cacert="/usr/local/etcd/ssl/ca.crt"  \--cert="/usr/local/etcd/ssl/server.crt" \--key="/usr/local/etcd/ssl/server.key" \--endpoints="https://192.168.44.186:2379,https://192.168.44.187:2379,https://192.168.44.186:2379"'
EOF
source /etc/profile

查看效果

[root@etcd-01 ~]# etcdctl endpoint status  --write-out=table
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|          ENDPOINT           |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://192.168.44.186:2379 | 21465eba80fe786a |  3.5.13 |   25 kB |     false |      false |         2 |         11 |                 11 |        |
| https://192.168.44.187:2379 | 5d144d32bb3de813 |  3.5.13 |   25 kB |     false |      false |         2 |         11 |                 11 |        |
| https://192.168.44.186:2379 | 21465eba80fe786a |  3.5.13 |   25 kB |     false |      false |         2 |         11 |                 11 |        |
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+