0%

在CentOS7上二进制部署Kubernetes高可用集群

注:本文参考360资深运维工程师李振良老师的《Kubernetes/K8S架构师实战训练营【中级班】》视频课程总结而成。部分内容和图片均来自于视频课程中。

在正式进行部署之前先了解下关于Kubernetes的几个核心知识点

Kubernetes是什么


  • Kubernetes是Google在2014年开源的一个容器集群管理系统,Kubernetes简称K8S。
  • K8S用于容器化应用程序的部署、扩展和管理
  • K8S提供了容器编排,资源调度,弹性伸缩,部署管理,服务发现等一系列功能。
  • Kubernetes目标是让部署容器化应用简单高效

Kubernetes特性


  • 自我修复

在节点故障时重新启动失败的容器,替换和重新部署,保证预期的副本数量;杀死健康检查失败的容器,并且在未准备好之前不会处理客户端请求,确保上线服务不中断。

  • 弹性伸缩

使用命令、UI或者基于CPU使用情况自动快速扩容和缩容应用程序实例,保证应用业务高峰并发时的高可用性;业务低峰时回收资源,以最小成本运行服务。

  • 自动部署和回滚

K8S采用滚动更新策略更新应用,一次更新一个Pod,而不是同时删除所有Pod,如果更新过程中出现问题,将回滚更改,确保升级不会影响业务。

  • 服务发现和负载均衡

K8S为多个容器提供一个统一访问入口(内部IP地址和一个DNS名称),并且负载均衡关联所有容器,使得用户无需考虑容器IP问题

  • 机密和配置管理

管理机密数据和应用程序配置,而不需要把敏感数据暴露在镜像里,提高敏感数据安全性。并可以将一些常用的配置存储在K8S中,方便应用程序使用。

  • 存储编排

挂载外部存储系统,无论是来自本地存储,公有云(如AWS),还是网络存储(如NFS、GlusterFS、Ceph)都作为集群资源的一部分使用,极大提高存储使用灵活性

  • 批处理

提供一次性任务,定时任务;满足批量数据处理和分析的场景

Kubernetes集群架构与组件


Master组件

  • kube-apiserver

Kubernetes API,集群的统一入口,各组件协调者,以RESTful API提供接口服务,所有对象资源的增删查改和监听操作都交给APIServer处理后再提交给etcd存储。

  • kube-controller-manager

处理集群中常规后台任务,一个资源对应一个控制器,而ControllerManager就是负责管理这些控制器的。

  • kube-scheduler

根据调度算法为新创建的Pod选择一个Node节点,可以任意部署,可以部署在同一个节点,也可以部署在不同的节点上。

  • etcd

分布式键值存储系统。用于保存集群状态数据,比如Pod、Service等对象信息。

Node组件

  • kubelet

kubelet是Master在Node节点上的Agent,管理本机运行容器的生命周期,比如创建容器、Pod挂载数据卷、下载secret、获取容器和节点状态等工作。kubelet将每个Pod转换成一组容器。

  • kube-proxy

在Node节点上实现Pod网络代理,维护网络规则和四层负载均衡工作

  • docker或rocket

容器引擎,运行容器

Kubernetes核心概念


Pod

  • 最小部署单元
  • 一组容器的集合
  • 一个Pod中的容器共享网络命名空间
  • Pod是暂短的

Controllers

  • ReplicaSet:确保预期的Pod副本数量
  • Deployment:无状态应用部署
  • StatefulSet:有状态应用部署
  • DaemonSet:确保所有Node运行同一个Pod
  • Job:一次性任务
  • CronJob:定时任务

Service

  • 防止Pod失联
  • 定义一组Pod的访问策略

Label

标签,附加到某个资源上,用于关联对象、查询和筛选

Namespace

命名空间,将对象逻辑上分离

生产环境K8S平台规划


单Master集群

多Master集群(HA)

服务器硬件配置推荐


Mater节点

  • 物理机虚拟机均可,至少1台,高可用集群至少2台(etcd集群必须奇数台)

  • 配置推荐:实验环境2核2G、测试环境2核4G、生产环境8核16G

  • 关闭所有swap分区或不划分swap分区

Node节点

  • 物理机虚拟机均可,大于等于1台

  • 配置推荐:实验环境2核2G、测试环境4核8G、生产环境16核64G

  • 关闭所有swap分区或不划分swap分区

实验环境信息


主机名 配置 操作系统 IP地址 角色 组件
k8s-master1 2核2G CentOS7.5 10.211.55.4 master kube-apiserver
kube-controller-manager
kube-scheduler
etcd
nginx
k8s-master2 2核2G CentOS7.5 10.211.55.7 master kube-apiserver
kube-controller-manager
kube-scheduler
nginx
k8s-node1 2核2G CentOS7.5 10.211.55.5 node kubelet
kube-proxy
docker
etcd
k8s-node2 2核2G CentOS7.5 10.211.55.6 node kubelet
kube-proxy
docker
etcd

VIP:10.211.55.10

系统初始化


关闭防火墙

1
2
systemctl stop firewalld
systemctl disable firewalld

关闭selinux

1
2
setenforce 0
sed -i "s/^SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config

关闭swap

1
2
swapoff -a
echo 'swapoff -a ' >> /etc/rc.d/rc.local

配置主机名

1
hostnamectl set-hostname <hostname>

添加所有节点的本地host解析

1
2
3
4
5
cat >> /etc/hosts << EOF
x.x.x.x hostname1
y.y.y.y hostname2
...
EOF

安装基础软件包

1
yum install vim net-tools lrzsz unzip dos2unix telnet sysstat iotop pciutils lsof tcpdump psmisc bc wget socat -y

内核开启网络支持

1
2
3
4
5
6
7
8
cat >  /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
net.ipv4.ip_nonlocal_bind = 1
EOF
modprobe br_netfilter
sysctl -p

配置所有master到所有节点(包括自身)的ssh免密登录

依此在所有的master节点上做如下操作:

1
2
ssh-keygen -t rsa
ssh-copy-id -i ~/.ssh/id_rsa.pub k8s-master1

节点之间时间同步

server端

注:如果环境可以访问互联网,可以不需要自己搭建server端,参考后面的client端部分设置所有节点与公网ntp时间服务器(例如time1.cloud.tencent.com)同步时间即可

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
#安装chrony并备份配置文件
yum install chrony ntpdate -y
cp -a /etc/chrony.conf /etc/chrony.conf.bak

#修改server端配置文件如下,标注的地方需要修改
cat > /etc/chrony.conf << EOF
stratumweight 0
driftfile /var/lib/chrony/drift
rtcsync
makestep 10 3
allow 10.211.55.0/24 #设置为实际环境客户端所属IP网段
smoothtime 400 0.01

bindcmdaddress 127.0.0.1
bindcmdaddress ::1

local stratum 8
manual
keyfile /etc/chrony.keys
#initstepslew 10 client1 client3 client6
noclientlog
logchange 0.5
logdir /var/log/chrony
EOF

#启动服务,设置开机自启
systemctl restart chronyd.service
systemctl enable chronyd.service
systemctl status chronyd.service

client端

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
#安装chrony并备份配置文件
yum install chrony ntpdate -y
cp -a /etc/chrony.conf /etc/chrony.conf.bak

#修改client端配置文件
sed -i "s%^server%#server%g" /etc/chrony.conf
echo "server 10.211.55.4 iburst" >> /etc/chrony.conf #添加一行,其中的IP地址替换为实际环境server端的IP地址

ntpdate 10.211.55.4 #手动同步一次时间,其中的IP地址替换为实际环境server端的IP地址

#启动服务,设置开机自启
systemctl restart chronyd.service
systemctl enable chronyd.service
systemctl status chronyd.service

chronyc sources #查看ntp_servers状态
chronyc tracking #查看ntp详细信息

安装CFSSL工具


CFSSL是CloudFlare开源的一款PKI/TLS工具。 CFSSL 包含一个命令行工具 和一个用于 签名,验证并且捆绑TLS证书的 HTTP API 服务。 使用Go语言编写

Github地址:https://github.com/cloudflare/cfssl

官网地址:https://pkg.cfssl.org/

在其中一台节点(一般是master1)上执行如下指令直接进行安装

1
2
3
4
curl -s -L -o /usr/local/bin/cfssl https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
curl -s -L -o /usr/local/bin/cfssljson https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
curl -s -L -o /usr/local/bin/cfssl-certinfo https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
chmod +x /usr/local/bin/cfssl*

如果环境无法联网,则到官网下载最新版本的cfssl_linux-amd64、cfssljson_linux-amd64、cfssl-certinfo_linux-amd64并上传到其中一台节点的/root目录下(一般是master1),并执行如下指令安装cfssl

1
2
3
4
mv cfssl_linux-amd64 /usr/local/bin/cfssl
mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
mv cfssl-certinfo_linux-amd64 /usr/local/bin/cfssl-certinfo
chmod +x /usr/local/bin/cfssl*

部署etcd数据库集群


etcd 是基于 Raft 的分布式 key-value 存储系统,由 CoreOS 开发,常用于服务发现、共享配置以及并发控制(如 leader 选举、分布式锁等)。kubernetes 使用 etcd 存储所有运行数据。

Github地址:https://github.com/etcd-io/etcd
官网地址:https://etcd.io/

使用cfssl为etcd生成自签证书

在安装了cfssl工具的节点上执行如下指令为etcd创建对应的ca机构并生成自签证书

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
#创建工作目录
mkdir /root/etcd-cert && cd /root/etcd-cert

#创建ca-csr.json文件

cat > ca-csr.json << EOF
{
"CN": "etcd CA",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "ShenZhen",
"ST": "ShenZhen"
}
]
}
EOF


#创建ca-config.json文件
#配置证书生成策略,让CA软件知道颁发什么样的证书
#ca-config.json:定义多个profiles,分别指定不同的过期时间,使用场景等参数,这里我们只定义了etcd一个profile
#signing:表示该证书可用于签名其它证书
#server auth:表示client可以使用该CA对server提供的证书进行验证
#client auth:表示server可以用该CA对client提供的证书进行验证

cat > ca-config.json << EOF
{
"signing": {
"default": {
"expiry": "876000h"
},
"profiles": {
"etcd": {
"expiry": "876000h",
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
]
}
}
}
}
EOF


#创建etcd-csr.json文件
#注:etcd-csr.json中的hosts字段需要所把有etcd集群节点的IP地址都添加进去

cat > etcd-csr.json << EOF
{
"CN": "etcd",
"hosts": [
"10.211.55.4",
"10.211.55.5",
"10.211.55.6"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "ShenZhen",
"ST": "ShenZhen"
}
]
}
EOF


#生成CA证书和私钥
cfssl gencert -initca ca-csr.json | cfssljson -bare ca -

#为etcd生成自签证书
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=etcd etcd-csr.json | cfssljson -bare etcd

部署etcd3.4之前版本

访问https://github.com/etcd-io/etcd/releases下载etcd3.4之前版本的二进制包(本文以3.3.18为例),并上传到其中一台etcd节点的/root目录下,然后执行如下指令解压并创建etcd相关目录和配置文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
#解压到/opt
cd /root/
tar zxf etcd-v3.3.18-linux-amd64.tar.gz -C /opt/
mv /opt/etcd-v3.3.18-linux-amd64/ /opt/etcd

#创建二进制文件存放目录、配置文件存放目录、证书存放目录
mkdir /opt/etcd/bin
mkdir /opt/etcd/cfg
mkdir /opt/etcd/ssl

#拷贝二进制文件到bin目录下
cp -a /opt/etcd/etcd* /opt/etcd/bin/

#拷贝上一步生成的自签证书到ssl目录下
cp -a /root/etcd-cert/{ca,etcd,etcd-key}.pem /opt/etcd/ssl/

#创建etcd集群配置文件,标注部分按实际情况进行修改
cat > /opt/etcd/cfg/etcd.conf << EOF
#[Member]
#自定义此etcd节点的名称,集群内唯一
ETCD_NAME="etcd-1"
#定义etcd数据存放目录
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
#定义本机和成员之间通信的地址
ETCD_LISTEN_PEER_URLS="https://10.211.55.4:2380"
#定义etcd对外提供服务的地址
ETCD_LISTEN_CLIENT_URLS="https://10.211.55.4:2379"

#[Clustering]
#定义该节点成员对等URL地址,且会通告集群的其余成员节点
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.211.55.4:2380"
#此成员的客户端URL列表,用于通告群集的其余部分
ETCD_ADVERTISE_CLIENT_URLS="https://10.211.55.4:2379"
#集群中所有节点的信息
ETCD_INITIAL_CLUSTER="etcd-1=https://10.211.55.4:2380,etcd-2=https://10.211.55.5:2380,etcd-3=https://10.211.55.6:2380"
#创建集群的token,这个值每个集群保持唯一
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
#设置new为初始静态或DNS引导期间出现的所有成员。如果将此选项设置为existing,则etcd将尝试加入现有群集
ETCD_INITIAL_CLUSTER_STATE="new"
EOF

#创建etcd的systemd unit文件
vim /usr/lib/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
EnvironmentFile=/opt/etcd/cfg/etcd.conf
ExecStart=/opt/etcd/bin/etcd \
--name=${ETCD_NAME} \
--data-dir=${ETCD_DATA_DIR} \
--listen-peer-urls=${ETCD_LISTEN_PEER_URLS} \
--listen-client-urls=${ETCD_LISTEN_CLIENT_URLS},http://127.0.0.1:2379 \
--advertise-client-urls=${ETCD_ADVERTISE_CLIENT_URLS} \
--initial-advertise-peer-urls=${ETCD_INITIAL_ADVERTISE_PEER_URLS} \
--initial-cluster=${ETCD_INITIAL_CLUSTER} \
--initial-cluster-token=${ETCD_INITIAL_CLUSTER_TOKEN} \
--initial-cluster-state=new \
--cert-file=/opt/etcd/ssl/etcd.pem \
--key-file=/opt/etcd/ssl/etcd-key.pem \
--peer-cert-file=/opt/etcd/ssl/etcd.pem \
--peer-key-file=/opt/etcd/ssl/etcd-key.pem \
--trusted-ca-file=/opt/etcd/ssl/ca.pem \
--peer-trusted-ca-file=/opt/etcd/ssl/ca.pem
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

将etcd目录和systemd unit文件拷贝到其余etcd集群节点上,并修改etcd配置文件中的名称IP地址

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
#拷贝etcd目录和etcd.service到其余etcd集群节点上
scp -r /opt/etcd/ k8s-node1:/opt/
scp -r /usr/lib/systemd/system/etcd.service k8s-node1:/usr/lib/systemd/system/

#修改etcd集群配置文件,标注部分按实际情况进行修改
vim /opt/etcd/cfg/etcd.conf
#[Member]
#自定义此etcd节点的名称,集群内唯一
ETCD_NAME="etcd-1"
#定义etcd数据存放目录
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
#定义本机和成员之间通信的地址
ETCD_LISTEN_PEER_URLS="https://10.211.55.4:2380"
#定义etcd对外提供服务的地址
ETCD_LISTEN_CLIENT_URLS="https://10.211.55.4:2379"

#[Clustering]
#定义该节点成员对等URL地址,且会通告集群的其余成员节点
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.211.55.4:2380"
#此成员的客户端URL列表,用于通告群集的其余部分
ETCD_ADVERTISE_CLIENT_URLS="https://10.211.55.4:2379"
#集群中所有节点的信息
ETCD_INITIAL_CLUSTER="etcd-1=https://10.211.55.4:2380,etcd-2=https://10.211.55.5:2380,etcd-3=https://10.211.55.6:2380"
#创建集群的token,这个值每个集群保持唯一
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
#设置new为初始静态或DNS引导期间出现的所有成员。如果将此选项设置为existing,则etcd将尝试加入现有群集
ETCD_INITIAL_CLUSTER_STATE="new"
EOF

在所有etcd集群节点上启动etcd并设置开机自启

1
2
3
4
5
6
systemctl daemon-reload
systemctl enable etcd
systemctl start etcd
#注:在第一台节点上执行start后会一直卡着无法返回命令提示符,这是因为在等待其他节点准备就绪,继续启动其余节点即可

systemctl status etcd

在任意etcd节点上执行如下指令查看集群状态,若所有节点均处于healthy状态则表示etcd集群部署成功

1
/opt/etcd/bin/etcdctl --ca-file=/opt/etcd/ssl/ca.pem --cert-file=/opt/etcd/ssl/etcd.pem --key-file=/opt/etcd/ssl/etcd-key.pem --endpoints="https://10.211.55.4:2379,https://10.211.55.5:2379,https://10.211.55.6:2379" cluster-health

部署etcd3.4版本

访问https://github.com/etcd-io/etcd/releases下载etcd3.4版本的二进制包(本文以3.4.3为例),并上传到其中一台etcd节点的/root目录下,然后执行如下指令解压并创建etcd相关目录和配置文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
#解压到/opt
cd /root/
tar zxf etcd-v3.4.3-linux-amd64.tar.gz -C /opt/
mv /opt/etcd-v3.4.3-linux-amd64/ /opt/etcd

#创建二进制文件存放目录、配置文件存放目录、证书存放目录
mkdir /opt/etcd/bin
mkdir /opt/etcd/cfg
mkdir /opt/etcd/ssl

#拷贝二进制文件到bin目录下
cp -a /opt/etcd/etcd* /opt/etcd/bin/

#拷贝上一步生成的自签证书到ssl目录下
cp -a /root/etcd-cert/{ca,etcd,etcd-key}.pem /opt/etcd/ssl/

#创建etcd集群配置文件,标注部分按实际情况进行修改
cat > /opt/etcd/cfg/etcd.conf << EOF
#[Member]
#自定义此etcd节点的名称,集群内唯一
ETCD_NAME="etcd-1"
#定义etcd数据存放目录
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
#定义本机和成员之间通信的地址
ETCD_LISTEN_PEER_URLS="https://10.211.55.4:2380"
#定义etcd对外提供服务的地址
ETCD_LISTEN_CLIENT_URLS="https://10.211.55.4:2379,http://127.0.0.1:2379"

#[Clustering]
#定义该节点成员对等URL地址,且会通告集群的其余成员节点
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.211.55.4:2380"
#此成员的客户端URL列表,用于通告群集的其余部分
ETCD_ADVERTISE_CLIENT_URLS="https://10.211.55.4:2379"
#集群中所有节点的信息
ETCD_INITIAL_CLUSTER="etcd-1=https://10.211.55.4:2380,etcd-2=https://10.211.55.5:2380,etcd-3=https://10.211.55.6:2380"
#创建集群的token,这个值每个集群保持唯一
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
#设置new为初始静态或DNS引导期间出现的所有成员。如果将此选项设置为existing,则etcd将尝试加入现有群集
ETCD_INITIAL_CLUSTER_STATE="new"
#flannel操作etcd使用的是v2的API,而kubernetes操作etcd使用的v3的API,在最新版ETCD3.4版本中默认关闭v2版本,所以为了兼容flannel,要设置开启v2的API
ETCD_ENABLE_V2="true"
EOF

#创建etcd的systemd unit文件
vim /usr/lib/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
EnvironmentFile=/opt/etcd/cfg/etcd.conf
ExecStart=/opt/etcd/bin/etcd \
--cert-file=/opt/etcd/ssl/etcd.pem \
--key-file=/opt/etcd/ssl/etcd-key.pem \
--peer-cert-file=/opt/etcd/ssl/etcd.pem \
--peer-key-file=/opt/etcd/ssl/etcd-key.pem \
--trusted-ca-file=/opt/etcd/ssl/ca.pem \
--peer-trusted-ca-file=/opt/etcd/ssl/ca.pem
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

将etcd目录和systemd unit文件拷贝到其余etcd集群节点上,并修改etcd配置文件中的名称IP地址

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
#拷贝etcd目录和etcd.service到其余etcd集群节点上
scp -r /opt/etcd/ k8s-node1:/opt/
scp -r /usr/lib/systemd/system/etcd.service k8s-node1:/usr/lib/systemd/system/

#修改etcd集群配置文件
vim /opt/etcd/cfg/etcd.conf
#[Member]
#自定义此etcd节点的名称,集群内唯一
ETCD_NAME="etcd-1"
#定义etcd数据存放目录
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
#定义本机和成员之间通信的地址
ETCD_LISTEN_PEER_URLS="https://10.211.55.4:2380"
#定义etcd对外提供服务的地址
ETCD_LISTEN_CLIENT_URLS="https://10.211.55.4:2379,http://127.0.0.1:2379"

#[Clustering]
#定义该节点成员对等URL地址,且会通告集群的其余成员节点
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.211.55.4:2380"
#此成员的客户端URL列表,用于通告群集的其余部分
ETCD_ADVERTISE_CLIENT_URLS="https://10.211.55.4:2379"
#集群中所有节点的信息
ETCD_INITIAL_CLUSTER="etcd-1=https://10.211.55.4:2380,etcd-2=https://10.211.55.5:2380,etcd-3=https://10.211.55.6:2380"
#创建集群的token,这个值每个集群保持唯一
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
#设置new为初始静态或DNS引导期间出现的所有成员。如果将此选项设置为existing,则etcd将尝试加入现有群集
ETCD_INITIAL_CLUSTER_STATE="new"
#flannel操作etcd使用的是v2的API,而kubernetes操作etcd使用的v3的API,在最新版ETCD3.4版本中默认关闭v2版本,所以为了兼容flannel,要设置开启v2的API
ETCD_ENABLE_V2="true"
EOF

在所有etcd集群节点上设置etcd开机自启并启动etcd

1
2
3
4
5
6
systemctl daemon-reload
systemctl enable etcd
systemctl start etcd
#注:在第一台节点上执行start后会一直卡着无法返回命令提示符,这是因为在等待其他节点准备就绪,继续启动其余节点即可

systemctl status etcd

在任意etcd节点上执行如下指令查看集群状态,若所有节点均处于healthy状态则表示etcd集群部署成功

1
/opt/etcd/bin/etcdctl --cacert=/opt/etcd/ssl/ca.pem --cert=/opt/etcd/ssl/etcd.pem --key=/opt/etcd/ssl/etcd-key.pem --endpoints="https://10.211.55.4:2379,https://10.211.55.5:2379,https://10.211.55.6:2379" endpoint health

卸载etcd

若安装失败需要卸载重新安装,在所有etcd节点上执行如下指令即可:

1
2
3
4
5
systemctl stop etcd
systemctl disable etcd
rm -rf /opt/etcd/
rm -rf /usr/lib/systemd/system/etcd.service
rm -rf /var/lib/etcd/

部署Master组件


使用cfssl为apiserver和kube-proxy生成自签证书

在安装了cfssl工具的节点上执行如下指令为apiserver和kube-proxy创建对应的ca机构并生成自签证书

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
#创建工作目录
mkdir /root/k8s-cert && cd /root/k8s-cert


#配置ca-csr.json

cat > ca-csr.json << EOF
{
"CN": "kubernetes",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "ShenZhen",
"ST": "ShenZhen",
"O": "k8s",
"OU": "System"
}
]
}
EOF


#创建ca-config.json文件
#配置证书生成策略,让CA软件知道颁发什么样的证书
#ca-config.json:定义多个profiles,分别指定不同的过期时间,使用场景等参数,这里我们只定义了etcd一个profile
#signing:表示该证书可用于签名其它证书
#server auth:表示client可以使用该CA对server提供的证书进行验证
#client auth:表示server可以用该CA对client提供的证书进行验证

cat > ca-config.json << EOF
{
"signing": {
"default": {
"expiry": "876000h"
},
"profiles": {
"kubernetes": {
"expiry": "876000h",
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
]
}
}
}
}
EOF


#创建apiserver-csr.json文件
#注:hosts字段需要把所有master节点、负载均衡节点的IP地址和VIP地址,还有规划的service-cluster-ip-range(在kube-apiserver.conf和kube-controller-manager.conf中配置)的第一个IP地址(本例中为10.0.0.1)都添加进去,其中的127.0.0.1和kubernetes.*部分不要修改
cat > apiserver-csr.json << EOF
{
"CN": "kubernetes",
"hosts": [
"10.0.0.1",
"127.0.0.1",
"kubernetes",
"kubernetes.default",
"kubernetes.default.svc",
"kubernetes.default.svc.cluster",
"kubernetes.default.svc.cluster.local",
"10.211.55.4",
"10.211.55.5",
"10.211.55.6",
"10.211.55.7",
"10.211.55.10"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "ShenZhen",
"ST": "ShenZhen",
"O": "k8s",
"OU": "System"
}
]
}
EOF


#创建kube-proxy-csr.json文件
cat > kube-proxy-csr.json << EOF
{
"CN": "system:kube-proxy",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "ShenZhen",
"ST": "ShenZhen",
"O": "k8s",
"OU": "System"
}
]
}
EOF


#生成CA证书和私钥
cfssl gencert -initca ca-csr.json | cfssljson -bare ca -

#为apiserver生成自签证书
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes apiserver-csr.json | cfssljson -bare apiserver

#为kube-proxy生成自签证书
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy

部署apiserver、controller-manager和scheduler

二进制包下载地址:https://github.com/kubernetes/kubernetes/releases

在每个release版本的CHANGELOG中有每个版本的二进制包下载列表,下载对应平台下的Server Binaries,上传到其中一台Master节点的/root目录下(本文以1.16.4为例)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
#创建kubernetes工作目录
mkdir /opt/kubernetes
mkdir /opt/kubernetes/bin
mkdir /opt/kubernetes/cfg
mkdir /opt/kubernetes/ssl
mkdir /opt/kubernetes/logs

#解压master binaries并拷贝所需的二进制文件到/opt/kubernetes/bin目录下
cd /root/
tar zxf kubernetes-server-linux-amd64.tar.gz
cp -a /root/kubernetes/server/bin/kube-apiserver /opt/kubernetes/bin/
cp -a /root/kubernetes/server/bin/kube-controller-manager /opt/kubernetes/bin/
cp -a /root/kubernetes/server/bin/kube-scheduler /opt/kubernetes/bin/
cp -a /root/kubernetes/server/bin/kubectl /usr/local/bin/

#拷贝自签证书到/opt/kubernetes/ssl目录下
cp -a /root/k8s-cert/{ca,ca-key,apiserver,apiserver-key}.pem /opt/kubernetes/ssl/

#创建kube-apiserver配置文件,标注部分按实际情况进行修改
vim /opt/kubernetes/cfg/kube-apiserver.conf
KUBE_APISERVER_OPTS="--logtostderr=false \
--v=2 \
--log-dir=/opt/kubernetes/logs \
--etcd-servers=https://10.211.55.4:2379,https://10.211.55.5:2379,https://10.211.55.6:2379 \
--bind-address=10.211.55.4 \
--secure-port=6443 \
--advertise-address=10.211.55.4 \
--allow-privileged=true \
--service-cluster-ip-range=10.0.0.0/24 \
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \
--authorization-mode=RBAC,Node \
--enable-bootstrap-token-auth=true \
--token-auth-file=/opt/kubernetes/cfg/token.csv \
--service-node-port-range=30000-32767 \
--kubelet-client-certificate=/opt/kubernetes/ssl/apiserver.pem \
--kubelet-client-key=/opt/kubernetes/ssl/apiserver-key.pem \
--tls-cert-file=/opt/kubernetes/ssl/apiserver.pem \
--tls-private-key-file=/opt/kubernetes/ssl/apiserver-key.pem \
--client-ca-file=/opt/kubernetes/ssl/ca.pem \
--service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \
--etcd-cafile=/opt/etcd/ssl/ca.pem \
--etcd-certfile=/opt/etcd/ssl/etcd.pem \
--etcd-keyfile=/opt/etcd/ssl/etcd-key.pem \
--audit-log-maxage=30 \
--audit-log-maxbackup=3 \
--audit-log-maxsize=100 \
--audit-log-path=/opt/kubernetes/logs/k8s-audit.log"

–logtostderr:日志是否输出到标准错误输出

–v=2:日志级别0-8,数字越大,日志越详细

–log-dir:设置日志存放目录

–etcd-servers:指定etcd服务的URL

–bind-address:apiserver监听地址

–secure-port:apiserver监听端口,默认为6443

–advertise-address:通告地址,让其他节点通过此IP来连接apiserver

–allow-privileged:开启容器的privileged权限

–service-cluster-ip-range:Kubernetes集群中Service的虚拟IP地址范围,以CIDR格式表示,例如169.169.0.0/16,该IP范围不能与部署机器的IP地址有重合

–enable-admission-plugins:Kubernetes集群的准入控制设置,各控制模块以插件的形式依次生效。

–authorization-mode:授权模式

–enable-bootstrap-token-auth:启用bootstrap token认证

–service-node-port-range:Kubernetes集群中Service可使用的端口号范围,默认值为30000~32767

–kubelet-client-certificate、–kubelet-client-key:连接kubelet使用的证书和私钥

–tls-cert-file、–tls-private-key-file、–client-ca-file、–service-account-key-file:apiserver启用https所用的证书和私钥

–etcd-cafile、–etcd-certfile、–etcd-keyfile:连接etcd所使用的证书

–audit-log-maxage、–audit-log-maxbackup、–audit-log-maxsize、–audit-log-path:日志轮转、日志路径

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
#创建kube-controller-manage配置文件,标注部分按实际情况进行修改
vim /opt/kubernetes/cfg/kube-controller-manager.conf
KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=false \
--v=2 \
--log-dir=/opt/kubernetes/logs \
--leader-elect=true \
--master=127.0.0.1:8080 \
--address=127.0.0.1 \
--allocate-node-cidrs=true \
--cluster-cidr=10.244.0.0/16 \
--service-cluster-ip-range=10.0.0.0/24 \
--cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \
--cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem \
--root-ca-file=/opt/kubernetes/ssl/ca.pem \
--service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem \
--experimental-cluster-signing-duration=876000h0m0s \
--node-monitor-grace-period=20s \
--node-monitor-period=2s \
--node-startup-grace-period=20s \
--pod-eviction-timeout=20s"

–leader-elect:启用自动选举

–master:连接apiserver的IP,127.0.0.1:8080是apiserver默认监听的,用于让其他组件通过此地址连接

–address:配置controller-manager监听地址,不需要对外

–allocate-node-cidrs:允许安装CNI插件,自动分配IP

–cluster-cidr:集群pod的IP段,要与与CNI插件的IP段一致

–service-cluster-ip-range:service cluster IP段,与apiserver中配置保持一致

–cluster-signing-cert-file、–cluster-signing-key-file:用于集群签名的ca证书和私钥

–root-ca-file、–service-account-private-key-file:签署service account的证书和私钥

–experimental-cluster-signing-duration:签发证书的有效期

1
2
3
4
5
6
7
8
#创建kube-scheduler配置文件,标注部分按实际情况进行修改
vim /opt/kubernetes/cfg/kube-scheduler.conf
KUBE_SCHEDULER_OPTS="--logtostderr=false \
--v=2 \
--log-dir=/opt/kubernetes/logs \
--leader-elect \
--master=127.0.0.1:8080 \
--address=127.0.0.1"

–leader-elect:启用自动选举

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
#创建kube-apiserver的systemd unit文件
vim /usr/lib/systemd/system/kube-apiserver.service
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-apiserver.conf
ExecStart=/opt/kubernetes/bin/kube-apiserver $KUBE_APISERVER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target



#创建kube-controller-manager的systemd unit文件
vim /usr/lib/systemd/system/kube-controller-manager.service
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-controller-manager.conf
ExecStart=/opt/kubernetes/bin/kube-controller-manager $KUBE_CONTROLLER_MANAGER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target



#创建kube-scheduler的systemd unit文件
vim /usr/lib/systemd/system/kube-scheduler.service
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-scheduler.conf
ExecStart=/opt/kubernetes/bin/kube-scheduler $KUBE_SCHEDULER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target

随机生成一个32位字符串,用以创建token.csv文件

1
2
3
token=`head -c 16 /dev/urandom | od -An -t x | tr -d ' '`
echo "$token,kubelet-bootstrap,10001,'system:node-bootstrapper'" > /opt/kubernetes/cfg/token.csv
#token.csv文件的格式为:(第一列)随机字符串,(第二列)用户名,(第三列)UID,(第四列)用户组

注:此处apiserver配置的token(32位随机字符串)必须要与后面node节点bootstrap.kubeconfig配置里的token一致

设置api-server、controller-manager、scheduler开机自启并启动

1
2
3
4
5
6
7
8
9
10
systemctl daemon-reload
systemctl enable kube-apiserver
systemctl enable kube-controller-manager
systemctl enable kube-scheduler
systemctl start kube-apiserver
systemctl start kube-controller-manager
systemctl start kube-scheduler
systemctl status kube-apiserver
systemctl status kube-controller-manager
systemctl status kube-scheduler

在其中一台master上执行如下指令为kubectl TLS Bootstrapping授权

1
kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap

此时你可以通过执行kubectl get cs获取k8s的各服务端组件状态看是否Healthy,但是如果你安装的是1.16版本,你会发现你的输出内容有一些变化,类似如下:

1
2
3
4
5
6
7
# kubectl get cs
NAME AGE
controller-manager <unknown>
scheduler <unknown>
etcd-0 <unknown>
etcd-2 <unknown>
etcd-1 <unknown>

起初可能会以为集群部署有问题,但是通过kubectl get cs -o yaml发现status、message等信息都有,只是没有打印出来。所以在网上搜索相关文章,最后在这里找到了原因,原来这是1.16的bug!!!坐等官方修复吧,当然针对此问题,大佬也给出了临时解决办法,参考下方:

1
kubectl get cs -o=go-template='{{printf "|NAME|STATUS|MESSAGE|\n"}}{{range .items}}{{$name := .metadata.name}}{{range .conditions}}{{printf "|%s|%s|%s|\n" $name .status .message}}{{end}}{{end}}'

部署Node组件


安装Docker

二进制包下载地址:https://download.docker.com/linux/static/stable/

到对应平台的目录下载所需版本的Docker二进制包,并上传到Node节点的/root目录下(本文以x86平台下的Docker18.09.9为例)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
#解压并拷贝二进制文件到对应目录下
cd /root/
tar zxf docker-18.09.9.tgz
cp -a docker/* /usr/bin/
chmod 755 /usr/bin/{containerd,containerd-shim,ctr,docker,dockerd,docker-init,docker-proxy,runc}

#创建docker的systemd unit文件
vim /etc/systemd/system/docker.service
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service containerd.service
Wants=network-online.target

[Service]
Type=notify
ExecStart=/usr/bin/dockerd
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always
StartLimitBurst=3
StartLimitInterval=60s
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
Delegate=yes
KillMode=process

[Install]
WantedBy=multi-user.target


#设置docker开机自启并启动
systemctl daemon-reload
systemctl start docker
systemctl enable docker
systemctl status docker

部署kubelet和kube-proxy

二进制包下载地址:https://github.com/kubernetes/kubernetes/releases

在每个release版本的CHANGELOG中有每个版本的二进制包下载列表,下载对应平台下的Node Binaries,上传到Node节点的/root目录下(本文以1.16.4为例)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
#创建kubernetes工作目录
mkdir /opt/kubernetes
mkdir /opt/kubernetes/bin
mkdir /opt/kubernetes/cfg
mkdir /opt/kubernetes/ssl
mkdir /opt/kubernetes/logs

#解压node binaries并拷贝所需的二进制文件到/opt/kubernetes/bin目录下
cd /root/
tar zxf kubernetes-node-linux-amd64.tar.gz
cp -a /root/kubernetes/node/bin/kubelet /opt/kubernetes/bin/
cp -a /root/kubernetes/node/bin/kube-proxy /opt/kubernetes/bin/

#创建kubelet.conf配置文件,标注部分按实际情况进行修改
#--hostname-override:当前节点注册到k8s显示的名称,集群内唯一
#--network-plugin:启用网络插件

vim /opt/kubernetes/cfg/kubelet.conf
KUBELET_OPTS="--logtostderr=false \
--v=2 \
--log-dir=/opt/kubernetes/logs \
--hostname-override=k8s-node1 \
--network-plugin=cni \
--kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \
--bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \
--config=/opt/kubernetes/cfg/kubelet-config.yml \
--cert-dir=/opt/kubernetes/ssl"



#创建bootstrap.kubeconfig配置文件
#server字段设置master节点的IP地址和端口
#token字段需要与master节点apiserver的token.csv文件中指定的token值一致

vim /opt/kubernetes/cfg/bootstrap.kubeconfig
apiVersion: v1
clusters:
- cluster:
certificate-authority: /opt/kubernetes/ssl/ca.pem
server: https://10.211.55.4:6443
name: kubernetes
contexts:
- context:
cluster: kubernetes
user: kubelet-bootstrap
name: default
current-context: default
kind: Config
preferences: {}
users:
- name: kubelet-bootstrap
user:
token: 65cc0bcbe77f4877f288e5604529f384



#创建kubelet-config.yml配置文件,
vim /opt/kubernetes/cfg/kubelet-config.yml
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
address: 0.0.0.0
port: 10250
readOnlyPort: 10255
cgroupDriver: cgroupfs
clusterDNS:
- 10.0.0.2
clusterDomain: cluster.local
failSwapOn: false
authentication:
anonymous:
enabled: false
webhook:
cacheTTL: 2m0s
enabled: true
x509:
clientCAFile: /opt/kubernetes/ssl/ca.pem
authorization:
mode: Webhook
webhook:
cacheAuthorizedTTL: 5m0s
cacheUnauthorizedTTL: 30s
evictionHard:
imagefs.available: 15%
memory.available: 100Mi
nodefs.available: 10%
nodefs.inodesFree: 5%
maxOpenFiles: 1000000
maxPods: 110



#创建kube-proxy.conf配置文件
vim /opt/kubernetes/cfg/kube-proxy.conf
KUBE_PROXY_OPTS="--logtostderr=false \
--v=2 \
--log-dir=/opt/kubernetes/logs \
--config=/opt/kubernetes/cfg/kube-proxy-config.yml"



#创建kube-proxy.kubeconfig配置文件
#server字段设置master节点的IP地址和端口
vim /opt/kubernetes/cfg/kube-proxy.kubeconfig
apiVersion: v1
clusters:
- cluster:
certificate-authority: /opt/kubernetes/ssl/ca.pem
server: https://10.211.55.4:6443
name: kubernetes
contexts:
- context:
cluster: kubernetes
user: kube-proxy
name: default
current-context: default
kind: Config
preferences: {}
users:
- name: kube-proxy
user:
client-certificate: /opt/kubernetes/ssl/kube-proxy.pem
client-key: /opt/kubernetes/ssl/kube-proxy-key.pem




#创建kube-proxy-config.yml配置文件
#hostnameOverride字段配置当前节点注册到k8s显示的名称,集群内唯一
#mode字段配置kube-proxy使用的模式,iptables or ipvs

vim /opt/kubernetes/cfg/kube-proxy-config.yml
kind: KubeProxyConfiguration
apiVersion: kubeproxy.config.k8s.io/v1alpha1
address: 0.0.0.0
metricsBindAddress: 0.0.0.0:10249
clientConnection:
kubeconfig: /opt/kubernetes/cfg/kube-proxy.kubeconfig
hostnameOverride: k8s-node1
clusterCIDR: 10.0.0.0/24
mode: ipvs
ipvs:
scheduler: "rr"
iptables:
masqueradeAll: true



#创建kubelet的systemd unit文件
vim /usr/lib/systemd/system/kubelet.service
[Unit]
Description=Kubernetes Kubelet
After=docker.service
Wants=docker.service

[Service]
EnvironmentFile=/opt/kubernetes/cfg/kubelet.conf
ExecStart=/opt/kubernetes/bin/kubelet $KUBELET_OPTS
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target



#创建kube-proxy的systemd unit文件
vim /usr/lib/systemd/system/kube-proxy.service
[Unit]
Description=Kubernetes Proxy
After=network.target

[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-proxy.conf
ExecStart=/opt/kubernetes/bin/kube-proxy $KUBE_PROXY_OPTS
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

从上面安装了cfssl工具的主机上拷贝ca证书和kube-proxy的自签证书和私钥到Node节点的/opt/kubernetes/ssl目录下

1
2
cd /root/k8s-cert/
scp -r ca.pem kube-proxy.pem kube-proxy-key.pem k8s-node1:/opt/kubernetes/ssl/

设置kubelet和kube-proxy开机自启并启动

1
2
3
4
5
6
7
systemctl daemon-reload
systemctl enable kubelet
systemctl enable kube-proxy
systemctl start kubelet
systemctl start kube-proxy
systemctl status kubelet
systemctl status kube-proxy

允许给Node颁发证书

当kubelet和kube-proxy成功启动后,此时在master上执行kubectl get csr可以看到有新的节点请求颁发证书(CONDITION字段处于Pending状态),执行如下指令允许给Node颁发证书

1
kubectl certificate approve node-csr-vCjAOsDYkXe4Af4gR-NBikSm4yIG00XV5zLjBZgzmQk

补充知识点

  • 若kubectl或kube-proxy配置文件中的hostname-override配置参数漏修改,导致授权后master无法正常获取到Node节点信息,除了修改kubelet.conf的–hostname-override配置和kube-proxy-config.yml的hostnameOverride配置外,还需要将kubelet.kubeconfig文件(这个文件是master认证后客户端自动生成的)删除,才可重新申请授权,否则报错信息类似如下:

    kubelet_node_status.go:94] Unable to register node “k8s-node2” with API server: nodes “k8s-node2” is forbidden: node “k8s-node1” is not allowed to modify node “k8s-node2”

  • TLS Bootstrapping 机制流程(Kubelet)

  • 如何删除一个Node节点并重新接入集群

在Master节点操作

1
2
kubectl drain 10.211.55.6 --delete-local-data
kubectl delete node 10.211.55.6

在Node节点操作

1
2
3
4
rm -rf /opt/kubernetes/cfg/kubelet.kubeconfig
rm -rf /opt/kubernetes/ssl/kubelet*
systemctl restart kubelet
systemctl restart kube-proxy

在Master节点重新授权

1
2
kubectl get csr
kubectl certificate approve xxxx

部署CNI网络

二进制下载地址:https://github.com/containernetworking/plugins/releases

下载所需平台的最新版本的CNI二进制包并上传到node节点的/root目录下

1
2
3
4
5
6
7
#创建cni的工作目录
mkdir -p /opt/cni/bin
mkdir -p /etc/cni/net.d

#解压到对应目录下
cd /root/
tar zxf cni-plugins-linux-amd64-v0.8.3.tgz -C /opt/cni/bin/

确保kubelet启用CNI

1
2
cat /opt/kubernetes/cfg/kubelet.conf |grep network-plugin
--network-plugin=cni \

在任意一台master节点执行如下指令下载网络yaml文件,k8s支持多种网络类型,本文安装的是flannel网络,更多网络类型可参考https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/#pod-network

1
2
cd /root/
wget https://raw.githubusercontent.com/coreos/flannel/2140ac876ef134e0ed5af15c65e414cf26827915/Documentation/kube-flannel.yml

根据实际环境情况修改kube-flannel.yml文件,比如Network、Backend、hostNetwork配置等等,修改完成后进行安装

1
2
cd /root/
kubectl apply -f kube-flannel.yml

查看flannel的pod创建情况,等待到所有node节点上的flannel pod都处于running状态后表示cni部署完成

1
kubectl get pods -n kube-system -o wide

授权apiserver访问kubelet

为提供安全性,kubelet禁止匿名访问,必须授权才可以。一个常见的表现就是无法通过kubectl logs查看pod的日志,错误输出类似如下:

Error from server (Forbidden): Forbidden (user=kubernetes, verb=get, resource=nodes, subresource=proxy) ( pods/log kube-flannel-ds-amd64-cdmcd)

在任意一台master节点上执行如下指令进行授权

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
#创建apiserver-to-kubelet-rbac.yaml文件
cat > /root/apiserver-to-kubelet-rbac.yaml << EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
annotations:
rbac.authorization.kubernetes.io/autoupdate: "true"
labels:
kubernetes.io/bootstrapping: rbac-defaults
name: system:kube-apiserver-to-kubelet
rules:
- apiGroups:
- ""
resources:
- nodes/proxy
- nodes/stats
- nodes/log
- nodes/spec
- nodes/metrics
- pods/log
verbs:
- "*"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: system:kube-apiserver
namespace: ""
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:kube-apiserver-to-kubelet
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: User
name: kubernetes
EOF


kubectl apply -f /root/apiserver-to-kubelet-rbac.yaml

环境测试验证

在任意一个master节点上执行如下指令创建一个nginx pod并暴露端口测试是否可以从外部正常访问

1
2
3
4
5
6
7
8
9
10
#创建nginx deployment
kubectl create deployment web --image=nginx

#暴露端口
kubectl expose deployment web --port=80 --type=NodePort

#查看对应的访问端口
kubectl get service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
web NodePort 10.0.0.200 <none> 80:30174/TCP 4s

浏览器访问:http://<Node_IP>:30174若能正常返回nginx欢迎页面,则表示环境一切正常。

部署Web UI和DNS


Web UI

Github地址:https://github.com/kubernetes/dashboard
官方地址:https://kubernetes.io/docs/tasks/access-application-cluster/web-ui-dashboard/

在任意一台master节点上下载Web UI的yaml文件到/root目录下

1
2
cd /root/
wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-beta6/aio/deploy/recommended.yaml

编辑recommended.yaml文件,找到kubernetes-dashboard这个Service的部分,设置其type为NodePort,nodePort为30001(可自定义)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
......

kind: Service
apiVersion: v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kubernetes-dashboard
spec:
type: NodePort
ports:
- port: 443
targetPort: 8443
nodePort: 30001
selector:
k8s-app: kubernetes-dashboard

......

修改完成后,执行如下指令开始部署Web UI

1
2
3
4
5
6
7
8
9
10
11
12
kubectl apply -f /root/recommended.yaml

#查看kubernetes-dashboard的pod,确认pod的STATUS均为running再继续下面的步骤
kubectl get pods -n kubernetes-dashboard
NAME READY STATUS RESTARTS AGE
dashboard-metrics-scraper-76585494d8-v4gd9 1/1 Running 0 63s
kubernetes-dashboard-b65488c4-pwkc2 1/1 Running 0 63s

#查看对应的service
kubectl get service -n kubernetes-dashboard
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes-dashboard NodePort 10.0.0.122 <none> 443:30001/TCP 2m45s

创建service account并绑定默认cluster-admin管理员集群角色

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
#创建授权的yaml文件
cat > /root/dashboard-adminuser.yaml << EOF
apiVersion: v1
kind: ServiceAccount
metadata:
name: admin-user
namespace: kubernetes-dashboard
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: admin-user
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: admin-user
namespace: kubernetes-dashboard
EOF

#执行授权
kubectl apply -f /root/dashboard-adminuser.yaml

获取token

1
kubectl -n kubernetes-dashboard describe secret $(kubectl -n kubernetes-dashboard get secret | grep admin-user | awk '{print $1}')

浏览器访问地址:https://<NODE_IP>:30001

使用上面输出的token登录dashboard,注意协议一定要用https

  • 注:若打开dashboard出现如下页面,没有”继续前往x.x.x.x(不安全)”的选项,则请参考下面的步骤进行处理

出现这个问题主要是因为之前版本部署的时候默认命名空间是kube-system,而新版的是kubernetes-dashboard导致,解决办法如下:

1.在刚执行指令部署Web UI的master节点上执行如下指令删除默认的secret,并用自签证书创建新的secret(注意修改自签证书的路径是否与实际环境一致)

1
2
kubectl delete secret kubernetes-dashboard-certs -n kubernetes-dashboard
kubectl create secret generic kubernetes-dashboard-certs --from-file=/opt/kubernetes/ssl/apiserver-key.pem --from-file=/opt/kubernetes/ssl/apiserver.pem -n kubernetes-dashboard

2.修改/root/recommended.yaml文件,在args下面增加证书两行(搜索auto-generate-certificates即可跳转到对应位置)

1
2
3
4
5
args:
# PLATFORM-SPECIFIC ARGS HERE
- --auto-generate-certificates
- --tls-key-file=apiserver-key.pem
- --tls-cert-file=apiserver.pem

3.重新应用recommended.yaml

1
kubectl apply -f /root/recommended.yaml

4.重新访问https://<NODE_IP>:30001,注意协议一定要用https

DNS

Github地址:https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/dns/coredns

部署DNS主要是为了给k8s的Service提供DNS解析服务,使得程序可以通过service的名称进行访问

  • DNS服务监视Kubernetes API,为每一个Service创建DNS记录用于域名解析。
  • ClusterIP A记录格式:<service-name>.<namespace-name>.svc.cluster.local,示例:my-svc.my-namespace.svc.cluster.local

使用kubeadm方式部署的k8s会自动安装CoreDNS,二进制部署方式则需要自行安装

从Github地址上下载coredns.yaml.base文件到任意master节点的/root/目录下,并重命名为coredns.yaml,然后参考下方标注修改其中的部分参数

  • __MACHINE_GENERATED_WARNING__替换为This is a file generated from the base underscore template file: coredns.yaml.base
  • __PILLAR__DNS__DOMAIN__替换为cluster.local,一般不修改,若要修改记得要与node节点上kubelet-config.yml文件中的clusterDomain的值一致,并要调整api-server证书中的hosts字段值并重新生产证书
  • __PILLAR__DNS__MEMORY__LIMIT__替换为170Mi,此内存限制的值可根据实际环境资源进行调整
  • __PILLAR__DNS__SERVER__替换为10.0.0.2,此IP地址需要与Node节点上/opt/kubernetes/cfg/kubelet-config.yml文件中配置的clusterDNS字段的IP一致

以下为我替换后最终的文件内容

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
# This is a file generated from the base underscore template file: coredns.yaml.base

apiVersion: v1
kind: ServiceAccount
metadata:
name: coredns
namespace: kube-system
labels:
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
kubernetes.io/bootstrapping: rbac-defaults
addonmanager.kubernetes.io/mode: Reconcile
name: system:coredns
rules:
- apiGroups:
- ""
resources:
- endpoints
- services
- pods
- namespaces
verbs:
- list
- watch
- apiGroups:
- ""
resources:
- nodes
verbs:
- get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
annotations:
rbac.authorization.kubernetes.io/autoupdate: "true"
labels:
kubernetes.io/bootstrapping: rbac-defaults
addonmanager.kubernetes.io/mode: EnsureExists
name: system:coredns
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:coredns
subjects:
- kind: ServiceAccount
name: coredns
namespace: kube-system
---
apiVersion: v1
kind: ConfigMap
metadata:
name: coredns
namespace: kube-system
labels:
addonmanager.kubernetes.io/mode: EnsureExists
data:
Corefile: |
.:53 {
errors
health {
lameduck 5s
}
ready
kubernetes cluster.local in-addr.arpa ip6.arpa {
pods insecure
fallthrough in-addr.arpa ip6.arpa
ttl 30
}
prometheus :9153
forward . /etc/resolv.conf
cache 30
loop
reload
loadbalance
}
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: coredns
namespace: kube-system
labels:
k8s-app: kube-dns
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
kubernetes.io/name: "CoreDNS"
spec:
# replicas: not specified here:
# 1. In order to make Addon Manager do not reconcile this replicas parameter.
# 2. Default is 1.
# 3. Will be tuned in real time if DNS horizontal auto-scaling is turned on.
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
selector:
matchLabels:
k8s-app: kube-dns
template:
metadata:
labels:
k8s-app: kube-dns
annotations:
seccomp.security.alpha.kubernetes.io/pod: 'runtime/default'
spec:
priorityClassName: system-cluster-critical
serviceAccountName: coredns
tolerations:
- key: "CriticalAddonsOnly"
operator: "Exists"
nodeSelector:
beta.kubernetes.io/os: linux
containers:
- name: coredns
image: k8s.gcr.io/coredns:1.6.5
imagePullPolicy: IfNotPresent
resources:
limits:
memory: 170Mi
requests:
cpu: 100m
memory: 70Mi
args: [ "-conf", "/etc/coredns/Corefile" ]
volumeMounts:
- name: config-volume
mountPath: /etc/coredns
readOnly: true
ports:
- containerPort: 53
name: dns
protocol: UDP
- containerPort: 53
name: dns-tcp
protocol: TCP
- containerPort: 9153
name: metrics
protocol: TCP
livenessProbe:
httpGet:
path: /health
port: 8080
scheme: HTTP
initialDelaySeconds: 60
timeoutSeconds: 5
successThreshold: 1
failureThreshold: 5
readinessProbe:
httpGet:
path: /ready
port: 8181
scheme: HTTP
securityContext:
allowPrivilegeEscalation: false
capabilities:
add:
- NET_BIND_SERVICE
drop:
- all
readOnlyRootFilesystem: true
dnsPolicy: Default
volumes:
- name: config-volume
configMap:
name: coredns
items:
- key: Corefile
path: Corefile
---
apiVersion: v1
kind: Service
metadata:
name: kube-dns
namespace: kube-system
annotations:
prometheus.io/port: "9153"
prometheus.io/scrape: "true"
labels:
k8s-app: kube-dns
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
kubernetes.io/name: "CoreDNS"
spec:
selector:
k8s-app: kube-dns
clusterIP: 10.0.0.2
ports:
- name: dns
port: 53
protocol: UDP
- name: dns-tcp
port: 53
protocol: TCP
- name: metrics
port: 9153
protocol: TCP

执行如下指令进行安装

1
2
3
4
kubectl apply -f /root/coredns.yaml

#确认dns相关的pod均为running状态
kubectl get pod -n kube-system
  • 测试验证

在任意master节点上执行如下指令创建一个busybox容器,在容器中ping service的名称看是否可以正常解析到IP地址并通信正常,如果可以则说明DNS服务部署成功。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
cat > /root/bs.yaml << EOF
apiVersion: v1
kind: Pod
metadata:
name: busybox
namespace: default
spec:
containers:
- image: busybox:1.28.4
command:
- sleep
- "3600"
imagePullPolicy: IfNotPresent
name: busybox
restartPolicy: Always
EOF

kubectl apply -f /root/bs.yaml
kubectl get pods

#待pod处于running状态后运行如下指令进入容器中
kubectl exec -ti busybox sh

#在容器中执行如下指令若出现类似输出则说明解析正常
/ # nslookup kubernetes
Server: 10.0.0.2
Address 1: 10.0.0.2 kube-dns.kube-system.svc.cluster.local

Name: kubernetes
Address 1: 10.0.0.1 kubernetes.default.svc.cluster.local
/ # ping kubernetes
PING kubernetes (10.0.0.1): 56 data bytes
64 bytes from 10.0.0.1: seq=0 ttl=64 time=0.042 ms
64 bytes from 10.0.0.1: seq=1 ttl=64 time=0.129 ms
^C
--- kubernetes ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.042/0.085/0.129 ms

到目前为止一套单Master+2个Node的K8S集群全部搭建完成,但是Master存在单点故障,因此在实际生产环境,我们需要部署多个Master节点,并在Master之前增加一层负载均衡(可通过Nginx、LVS、HAproxy实现),同时为了避免负载均衡存在单点故障,通过Keepalived来实现负载均衡的主备,这样就能保证整个集群不存在单点故障,所有组件均有高可用。

接下来我们将扩容一台Master节点,并通过Nginx和Keepalived实现高可用的负载均衡。

Master高可用


部署Master组件(同Master1一致)

新Master节点的系统初始化操作请参考之前章节进行,此处不再赘述

将Master1上的kubernetes工作目录、etc工作目录下的ssl目录(证书和私钥文件)、Master组件的systemd unit文件和kubectl二进制文件拷贝到新增Master节点的对应目录下

1
2
3
4
5
6
scp -r /opt/kubernetes/ k8s-master2:/opt/
ssh k8s-master2 rm -rf /opt/kubernetes/logs/*
ssh k8s-master2 mkdir /opt/etcd
scp -r /opt/etcd/ssl/ k8s-master2:/opt/etcd/
scp -r /usr/lib/systemd/system/{kube-apiserver,kube-controller-manager,kube-scheduler}.service k8s-master2:/usr/lib/systemd/system/
scp -r /usr/local/bin/kubectl k8s-master2:/usr/local/bin/

修改新Master节点上apiserver配置文件中的–bind-address和–advertise-address参数为本机IP

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
vim /opt/kubernetes/cfg/kube-apiserver.conf
KUBE_APISERVER_OPTS="--logtostderr=false \
--v=2 \
--log-dir=/opt/kubernetes/logs \
--etcd-servers=https://10.211.55.4:2379,https://10.211.55.5:2379,https://10.211.55.6:2379 \
--bind-address=10.211.55.7 \
--secure-port=6443 \
--advertise-address=10.211.55.7 \
--allow-privileged=true \
--service-cluster-ip-range=10.0.0.0/24 \
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \
--authorization-mode=RBAC,Node \
--enable-bootstrap-token-auth=true \
--token-auth-file=/opt/kubernetes/cfg/token.csv \
--service-node-port-range=30000-32767 \
--kubelet-client-certificate=/opt/kubernetes/ssl/apiserver.pem \
--kubelet-client-key=/opt/kubernetes/ssl/apiserver-key.pem \
--tls-cert-file=/opt/kubernetes/ssl/apiserver.pem \
--tls-private-key-file=/opt/kubernetes/ssl/apiserver-key.pem \
--client-ca-file=/opt/kubernetes/ssl/ca.pem \
--service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \
--etcd-cafile=/opt/etcd/ssl/ca.pem \
--etcd-certfile=/opt/etcd/ssl/etcd.pem \
--etcd-keyfile=/opt/etcd/ssl/etcd-key.pem \
--audit-log-maxage=30 \
--audit-log-maxbackup=3 \
--audit-log-maxsize=100 \
--audit-log-path=/opt/kubernetes/logs/k8s-audit.log"

设置api-server、controller-manager、scheduler开机自启并启动

1
2
3
4
5
6
7
8
9
10
systemctl daemon-reload
systemctl enable kube-apiserver
systemctl enable kube-controller-manager
systemctl enable kube-scheduler
systemctl start kube-apiserver
systemctl start kube-controller-manager
systemctl start kube-scheduler
systemctl status kube-apiserver
systemctl status kube-controller-manager
systemctl status kube-scheduler

在新Master节点上执行如下指令若能正常获取到node节点信息说明新Master节点新增成功

1
kubectl get nodes

部署Nginx负载均衡

Nginx RPM下载地址:http://nginx.org/packages/rhel/7/x86_64/RPMS/

下载Nginx安装包并上传到规划部署Nginx的机器的/root目录下,并进行安装

1
2
cd /root/
rpm -ivh nginx-1.16.1-1.el7.ngx.x86_64.rpm

配置Nginx配置文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
vim /etc/nginx/nginx.conf
......
events {
worker_connections 1024;
}

#新增如下stream部分
#upstream中依此列出所有master节点的IP:Port
#listen字段如果Nginx是和apiserver部署在同一台服务器上,需要使用非6443端口(本文使用8443),否则会产生端口冲突,若不是部署在同一台机器上则可以使用默认6443端口
stream {

log_format main '$remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent';

access_log /var/log/nginx/k8s-access.log main;

upstream k8s-apiserver {
server 10.211.55.4:6443;
server 10.211.55.7:6443;
}

server {
listen 8443;
proxy_pass k8s-apiserver;
}
}

http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
......

设置Nginx开机自启并启动

1
2
3
systemctl enable nginx
systemctl start nginx
systemctl status nginx

部署Keepalived

主节点

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
yum install keepalived -y
cp -a /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.bak

#配置keepalived配置文件
cat > /etc/keepalived/keepalived.conf << EOF
global_defs {
notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id NGINX_MASTER
}

vrrp_script check_nginx {
script "/etc/keepalived/check_nginx.sh"
}

vrrp_instance VI_1 {
state MASTER
interface eth0 # 接口名称
virtual_router_id 51 # VRRP 路由 ID实例,每个实例是唯一的
priority 100 # 优先级,备服务器设置 90
advert_int 1 # 指定VRRP 心跳包通告间隔时间,默认1秒
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
10.211.55.10/24
}
track_script {
check_nginx
}
}
EOF

#创建Nginx检测脚本
vim /etc/keepalived/check_nginx.sh
#!/bin/bash
count=$(ps -ef |grep nginx |egrep -cv "grep|$$")

if [ "$count" -eq 0 ];then
exit 1
else
exit 0
fi

#授予Nginx检测脚本可执行权限
chmod +x /etc/keepalived/check_nginx.sh

设置Keepalived开机自启并启动

1
2
3
systemctl enable keepalived
systemctl start keepalived
systemctl status keepalived

备节点

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
yum install keepalived -y
cp -a /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.bak

#配置keepalived配置文件
cat > /etc/keepalived/keepalived.conf << EOF
global_defs {
notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id NGINX_BACKUP
}

vrrp_script check_nginx {
script "/etc/keepalived/check_nginx.sh"
}

vrrp_instance VI_1 {
state BACKUP
interface eth0 # 接口名称
virtual_router_id 51 # VRRP 路由 ID实例,每个实例是唯一的
priority 90 # 优先级,备服务器设置 90
advert_int 1 # 指定VRRP 心跳包通告间隔时间,默认1秒
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
10.211.55.10/24
}
track_script {
check_nginx
}
}
EOF

#创建Nginx检测脚本
vim /etc/keepalived/check_nginx.sh
#!/bin/bash
count=$(ps -ef |grep nginx |egrep -cv "grep|$$")

if [ "$count" -eq 0 ];then
exit 1
else
exit 0
fi

#授予Nginx检测脚本可执行权限
chmod +x /etc/keepalived/check_nginx.sh

设置Keepalived开机自启并启动

1
2
3
systemctl enable keepalived
systemctl start keepalived
systemctl status keepalived

修改Node连接VIP

修改所有node节点上k8s的bootstrap.kubeconfig、kubelet.kubeconfig和kube-proxy.kubeconfig配置文件中的server字段的IP和Port信息,IP替换为VIP、Port替换为Nginx中配置的监听端口,然后重启kubelet和kube-proxy服务

1
2
3
4
5
6
sed -i "s/10.211.55.4:6443/10.211.55.10:8443/g" /opt/kubernetes/cfg/*
systemctl restart kubelet
systemctl restart kube-proxy

#确认node节点是否处于Ready状态
kubectl get nodes

测试VIP是否正常工作

在任意节点上执行如下指令调用API看是否可以正常查看版本信息。其中token替换为token.csv中的token值,IP替换为VIP,Port替换为Nginx中配置的监听端口

若VIP可以正常工作,可以尝试关闭其中一台Nginx,确认VIP是否可以正常漂移到backup节点,然后再次测试调用API是否正常,验证是否可以达到故障切换的效果

1
2
3
4
5
6
7
8
9
10
11
12
13
curl -k --header "Authorization: Bearer 65cc0bcbe77f4877f288e5604529f384" https://10.211.55.10:8443/version
#返回类似如下输出一切正常
{
"major": "1",
"minor": "16",
"gitVersion": "v1.16.4",
"gitCommit": "224be7bdce5a9dd0c2fd0d46b83865648e2fe0ba",
"gitTreeState": "clean",
"buildDate": "2019-12-11T12:37:43Z",
"goVersion": "go1.12.12",
"compiler": "gc",
"platform": "linux/amd64"
}

至此一套高可用架构的Kubernetes集群就部署完成了,架构图如下所示。若还需要再安装Helm、ingress-nginx、配置k8s对接外部存储做持久化存储,比如Ceph,可继续参考如下内容。

安装helm


helm官网:https://helm.sh/

helm github:https://github.com/helm/helm

下载所需版本的helm安装包(本文以2.16.1版本为例),上传到所有的master节点的/root/helm目录下(若没有此目录先创建),执行如下指令安装helm客户端

1
2
3
4
cd /root/helm
tar zxf helm-v2.16.1-linux-amd64.tar.gz
cd linux-amd64/
cp -a helm /usr/local/bin/

在其中一台master运行如下指令安装helm服务端

1
helm init

执行如下指令设置tiller的rbac权限

1
2
3
4
5
6
7
8
9
kubectl create serviceaccount -n kube-system tiller
kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
kubectl --namespace kube-system patch deploy tiller-deploy -p '{"spec":{"template":{"spec":{"serviceAccount":"tiller"}}}}'
helm init --upgrade --service-account tiller

#稍等片刻待tiller的pod处于均Ready后执行helm list看是否可以正常列出所有的release
kubectl get pods -n kube-system |grep tiller
helm list
helm version

若执行helm version出现类似如下报错

1
2
Client: &version.Version{SemVer:"v2.16.1", GitCommit:"bbdfe5e7803a12bbdf97e94cd847859890cf4050", GitTreeState:"clean"}
E1213 15:58:40.605638 10274 portforward.go:400] an error occurred forwarding 34583 -> 44134: error forwarding port 44134 to pod 1e92153b279110f9464193c4ea7d6314ac69e70ce60e7319df9443e379b52ed4, uid : unable to do port forwarding: socat not found

解决办法:

在所有node节点上安装socat

1
yum install socat -y

安装ingress-nginx


ingress-nginx官网:https://kubernetes.github.io/ingress-nginx/

ingress-nginx github:https://github.com/kubernetes/ingress-nginx

在任意一台master节点上下载ingress-nginx的yaml文件到/root目录下

1
2
cd /root/
wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/mandatory.yaml

编辑mandatory.yaml文件,设置ingress-nginx的部署模式,本文采用hostNetwork模式

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
...
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-ingress-controller
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
template:
metadata:
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
annotations:
prometheus.io/port: "10254"
prometheus.io/scrape: "true"
spec:
hostNetwork: true #添加此行设置ingress-nginx的部署模式为hostNetwork
# wait up to five minutes for the drain of connections
terminationGracePeriodSeconds: 300
serviceAccountName: nginx-ingress-serviceaccount
nodeSelector:
kubernetes.io/os: linux
...

运行如下指令安装ingress-nginx

1
2
3
4
5
cd /root/
kubectl apply -f mandatory.yaml

#确认ingress-nginx的pod是否正常启动,处于running状态
kubectl get pods -n ingress-nginx

配置rbd-provisioner


K8S要对接Ceph RBD存储做持久化存储,首先必须要搭建Ceph存储集群,并在K8S的所有节点上安装对应版本的ceph-common客户端命令。关于Ceph集群的搭建和ceph-common此处不进行赘述,可参考Ceph官网文档进行。

Ceph集群和ceph-common安装都完成后,在其中一台master上创建/root/rbd-provisioner目录下,并执行如下指令创建rbd-provisioner所需的yaml文件,标注部分根据实际情况进行修改

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
mkdir /root/rbd-provisioner
cd /root/rbd-provisioner

cat > clusterrole.yaml << EOF
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: rbd-provisioner
rules:
- apiGroups: [""]
resources: ["persistentvolumes"]
verbs: ["get", "list", "watch", "create", "delete"]
- apiGroups: [""]
resources: ["persistentvolumeclaims"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["events"]
verbs: ["create", "update", "patch"]
- apiGroups: [""]
resources: ["services"]
resourceNames: ["kube-dns","coredns"]
verbs: ["list", "get"]
- apiGroups: [""]
resources: ["endpoints"]
verbs: ["get", "list", "watch", "create", "update", "patch"]
EOF


cat > clusterrolebinding.yaml << EOF
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: rbd-provisioner
subjects:
- kind: ServiceAccount
name: rbd-provisioner
namespace: ceph
roleRef:
kind: ClusterRole
name: rbd-provisioner
apiGroup: rbac.authorization.k8s.io
EOF

cat > deployment.yaml << EOF
apiVersion: apps/v1
kind: Deployment
metadata:
name: rbd-provisioner
namespace: ceph
spec:
progressDeadlineSeconds: 600
revisionHistoryLimit: 10
replicas: 1
selector:
matchLabels:
app: rbd-provisioner
strategy:
type: Recreate
template:
metadata:
labels:
app: rbd-provisioner
spec:
containers:
- name: rbd-provisioner
imagePullPolicy: IfNotPresent
image: "quay.io/external_storage/rbd-provisioner:latest"
env:
- name: PROVISIONER_NAME
value: ceph.com/rbd
serviceAccount: rbd-provisioner
restartPolicy: Always
EOF

cat > role.yaml << EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: rbd-provisioner
namespace: ceph
rules:
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get"]
- apiGroups: [""]
resources: ["endpoints"]
verbs: ["get", "list", "watch", "create", "update", "patch"]
EOF


cat > rolebinding.yaml << EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: rbd-provisioner
namespace: ceph
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: rbd-provisioner
subjects:
- kind: ServiceAccount
name: rbd-provisioner
namespace: ceph
EOF

cat > serviceaccount.yaml << EOF
apiVersion: v1
kind: ServiceAccount
metadata:
name: rbd-provisioner
namespace: ceph
EOF


cat > storageclass.yaml << EOF
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
annotations:
storageclass.beta.kubernetes.io/is-default-class: "true"
name: rbd
provisioner: ceph.com/rbd
parameters:
monitors: 10.211.55.4:6789,10.211.55.5:6789,10.211.55.6:6789 #配置Ceph集群的monitor节点信息
pool: k8s #配置要连接的pool,若没有需要先在ceph集群上创建
adminId: admin
adminSecretNamespace: ceph
adminSecretName: ceph-secret
fsType: ext4
userId: admin
userSecretNamespace: ceph
userSecretName: ceph-secret
imageFormat: "2"
imageFeatures: layering
reclaimPolicy: Delete
volumeBindingMode: Immediate
EOF

cat > secrets.yaml << EOF
apiVersion: v1
kind: Secret
metadata:
name: ceph-secret
namespace: ceph
type: "ceph.com/rbd"
data:
# ceph auth add client.kube mon 'allow r' osd 'allow rwx pool=kube'
# ceph auth get-key client.admin | base64
key: QVFEcTN5VmRvK28xRHhBQUlKNW5zQ0xwcTd3N0Q5OTJENm9YeGc9PQ== #配置Ceph集群的kering,此处填的是经过base64编码后的值
EOF

执行如下执行创建rbd storageclass

1
2
3
4
cd /root/rbd-provisioner
kubectl create namespace ceph
kubectl apply -f storageclass.yaml -f clusterrolebinding.yaml -f clusterrole.yaml -f deployment.yaml -f rolebinding.yaml -f role.yaml -f secrets.yaml -f serviceaccount.yaml
kubectl get pods -n ceph | grep rbd-provisioner

配置cephfs-provisioner


K8S要对接CephFS存储做持久化存储,首先必须要搭建Ceph存储集群,并在K8S的所有节点上安装对应版本的ceph-common客户端命令。关于Ceph集群的搭建和ceph-common此处不进行赘述,可参考Ceph官网文档进行。

Ceph集群和ceph-common安装都完成后,在其中一台master上创建/root/cephfs-provisioner目录下,并执行如下指令创建cephfs-provisioner所需的yaml文件,标注部分根据实际情况进行修改

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
mkdir /root/cephfs-provisioner
cd /root/cephfs-provisioner

cat > clusterrole.yaml << EOF
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: cephfs-provisioner
namespace: ceph
rules:
- apiGroups: [""]
resources: ["persistentvolumes"]
verbs: ["get", "list", "watch", "create", "delete"]
- apiGroups: [""]
resources: ["persistentvolumeclaims"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["events"]
verbs: ["create", "update", "patch"]
- apiGroups: [""]
resources: ["services"]
resourceNames: ["kube-dns","coredns"]
verbs: ["list", "get"]
EOF

cat > clusterrolebinding.yaml << EOF
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: cephfs-provisioner
subjects:
- kind: ServiceAccount
name: cephfs-provisioner
namespace: ceph
roleRef:
kind: ClusterRole
name: cephfs-provisioner
apiGroup: rbac.authorization.k8s.io
EOF


cat > deployment.yaml << EOF
apiVersion: apps/v1
kind: Deployment
metadata:
name: cephfs-provisioner
namespace: ceph
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app: cephfs-provisioner
strategy:
type: Recreate
template:
metadata:
labels:
app: cephfs-provisioner
spec:
containers:
- name: cephfs-provisioner
image: "quay.io/external_storage/cephfs-provisioner:latest"
imagePullPolicy: IfNotPresent
env:
- name: PROVISIONER_NAME
value: ceph.com/cephfs
- name: PROVISIONER_SECRET_NAMESPACE
value: ceph
command:
- "/usr/local/bin/cephfs-provisioner"
args:
- "-id=cephfs-provisioner-1"
- "-disable-ceph-namespace-isolation=true"
- "-enable-quota=true"
serviceAccount: cephfs-provisioner
restartPolicy: Always
terminationGracePeriodSeconds: 30
EOF

cat > role.yaml << EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: cephfs-provisioner
namespace: ceph
rules:
- apiGroups: [""]
resources: ["secrets"]
verbs: ["create", "get", "delete"]
- apiGroups: [""]
resources: ["endpoints"]
verbs: ["get", "list", "watch", "create", "update", "patch"]
EOF


cat > rolebinding.yaml << EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: cephfs-provisioner
namespace: ceph
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: cephfs-provisioner
subjects:
- kind: ServiceAccount
name: cephfs-provisioner
EOF


cat > serviceaccount.yaml << EOF
apiVersion: v1
kind: ServiceAccount
metadata:
name: cephfs-provisioner
namespace: ceph
EOF


cat > storageclass.yaml << EOF
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: cephfs
provisioner: ceph.com/cephfs
parameters:
monitors: 10.211.55.4:6789,10.211.55.5:6789,10.211.55.6:6789 #配置Ceph集群的monitor节点信息
adminId: admin
adminSecretName: ceph-secret
adminSecretNamespace: "ceph"
reclaimPolicy: Delete
volumeBindingMode: Immediate
EOF

执行如下指令创建cephfs storageclass

1
2
3
cd /root/cephfs-provisioner
kubectl apply -f storageclass.yaml -f clusterrolebinding.yaml -f clusterrole.yaml -f deployment.yaml -f rolebinding.yaml -f role.yaml -f serviceaccount.yaml
kubectl get pods -n ceph | grep cephfs-provisioner

参考链接:

https://blog.csdn.net/snipercai/article/details/101012124

https://segmentfault.com/a/1190000020912684


- - - - - - - - - 本文结束啦感谢您阅读 - - - - - - - - -