1.准备
| IP | 系统 | 角色 |
| 192.168.123.21 | Centos7 | master |
| 192.168.123.23 | Centos7 | node |
| 192.168.123.25 | Centos7 | node |
* 以下所有操作全是在代理后面完成的,如果没有代理可能无法搭建成功.
1.1 基本配置
1.1.1 设置主机名
在每台机器上执行,换乘对应名字
hostnamectl set-hostname k8s-master hostnamectl set-hostname k8s-node1 hostnamectl set-hostname k8s-node2
cat >> /etc/hosts <<EOF 192.168.123.21 k8s-master 192.168.123.23 k8s-node1 192.168.123.25 k8s-node2 EOF
1.1.2 SSH无密码登录其它机器
ssh-keygen -t rsa ssh-copy-id root@k8s-master ssh-copy-id root@k8s-node1 ssh-copy-id root@k8s-node2
1.1.3 升级内核
CentOS 7.x 系统自带的 3.10.x 内核存在一些 Bugs，导致运行的 Docker、Kubernetes 不稳定,所以把内核升级到4.4.X 以上；
rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-3.el7.elrepo.noarch.rpm # 安装完成后检查 /boot/grub2/grub.cfg 中对应内核 menuentry 中是否包含 initrd16 配置，如果没有，再安装一次！ yum --enablerepo=elrepo-kernel install -y kernel-lt # 设置开机从新内核启动 grub2-set-default 0
内核安装完成需要重启机器
安装内核源文件（可选，在升级完内核并重启机器后执行）:
# yum erase kernel-headers yum --enablerepo=elrepo-kernel install kernel-lt-devel-$(uname -r) kernel-lt-headers-$(uname -r)
1.1.4 内核参数优化
cat > /etc/sysctl.d/k8s.conf <<EOF net.bridge.bridge-nf-call-iptables=1 net.bridge.bridge-nf-call-ip6tables=1 net.ipv4.ip_forward=1 net.ipv4.tcp_tw_recycle=0 vm.swappiness=0 # 禁止使用 swap 空间，只有当系统 OOM 时才允许使用它 vm.overcommit_memory=1 # 不检查物理内存是否够用 vm.panic_on_oom=0 # 开启 OOM fs.inotify.max_user_instances=8192 fs.inotify.max_user_watches=1048576 fs.file-max=52706963 fs.nr_open=52706963 net.ipv6.conf.all.disable_ipv6=1 net.netfilter.nf_conntrack_max=2310720 EOF modprobe br_netfilter && sysctl -p /etc/sysctl.d/k8s.conf
1.1.5 加载内核模块
cat > /etc/sysconfig/modules/ipvs.modules <<EOF #!/bin/bashmodprobe -- ip_vs modprobe -- ip_vs_rr modprobe -- ip_vs_wrr modprobe -- ip_vs_sh modprobe -- nf_conntrack_ipv4 EOF chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack_ipv4
1.1.6 关闭 swap 分区
如果机器上有其他的服务依赖swap,可以跳过.
下面for循环的代码只需要在master机器上执行就可以同步到其他机器上了,但是首先要执行一次
export NODE_IPS=(192.168.123.21 192.168.123.23 192.168.123.25)
for node_ip in ${NODE_IPS[@]}
do
echo ">>> ${node_ip}"
ssh root@${node_ip} "swapoff -a && sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab"
done
1.1.7 关闭 SELinux
for node_ip in ${NODE_IPS[@]}
do
echo ">>> ${node_ip}"
ssh root@${node_ip} "setenforce 0 && sed -i 's/^SELINUX=.*/SELINUX=disabled/' /etc/selinux/config"
done
1.1.8 关闭防火墙,并设置iptables filter表中FOWARD链的默认策略(pllicy)为ACCEPT。
for node_ip in ${NODE_IPS[@]}
do
echo ">>> ${node_ip}"
ssh root@${node_ip} "systemctl stop firewalld && systemctl disable firewalld && iptables -P FORWARD ACCEPT"
done
1.1.9 设置系统时区
for node_ip in ${NODE_IPS[@]}
do
echo ">>> ${node_ip}"
ssh root@${node_ip} "timedatectl set-timezone Asia/Shanghai"
done
1.2 安装Docker
Kubernetes从1.6开始使用CRI(Container Runtime Interface)容器运行时接口。默认的容器运行时仍然是Docker，使用的是kubelet中内置dockershim CRI实现。
安装docker的yum源:
yum install -y yum-utils device-mapper-persistent-data lvm2 yum-config-manager \ --add-repo \ https://download.docker.com/linux/centos/docker-ce.repo
查看最新的Docker版本：
yum list docker-ce.x86_64 --showduplicates |sort -r docker-ce.x86_64 3:18.09.7-3.el7 docker-ce-stable docker-ce.x86_64 3:18.09.6-3.el7 docker-ce-stable docker-ce.x86_64 3:18.09.5-3.el7 docker-ce-stable docker-ce.x86_64 3:18.09.4-3.el7 docker-ce-stable docker-ce.x86_64 3:18.09.3-3.el7 docker-ce-stable docker-ce.x86_64 3:18.09.2-3.el7 docker-ce-stable docker-ce.x86_64 3:18.09.1-3.el7 docker-ce-stable docker-ce.x86_64 3:18.09.0-3.el7 docker-ce-stable docker-ce.x86_64 18.06.3.ce-3.el7 docker-ce-stable docker-ce.x86_64 18.06.2.ce-3.el7 docker-ce-stable docker-ce.x86_64 18.06.1.ce-3.el7 docker-ce-stable docker-ce.x86_64 18.06.0.ce-3.el7 docker-ce-stable docker-ce.x86_64 18.03.1.ce-1.el7.centos docker-ce-stable docker-ce.x86_64 18.03.0.ce-1.el7.centos docker-ce-stable
Kubernetes 1.15当前支持的docker版本列表是1.13.1, 17.03, 17.06, 17.09, 18.06, 18.09。 这里在各节点安装docker的18.09.7版本。
yum install -y docker-ce-18.09.7-3.el7 systemctl start docker && systemctl enable docker
修改docker 的配置,native.cgroupdriver=systemd 此项必须修改.
cat << EOF > /etc/docker/daemon.json
{
"registry-mirrors":[
"https://c6ai9izk.mirror.aliyuncs.com"
],
"log-driver":"json-file",
"log-opts":{
"max-size":"100m"
},
"storage-driver":"overlay2",
"storage-opts": [
"overlay2.override_kernel_check=true"
],
"live-restore": true,
"exec-opts": [
"native.cgroupdriver=systemd"
],
"insecure-registries": []
}
EOF
systemctl restart docker注：daemon.json解析
{
"authorization-plugins": [], //访问授权插件
"data-root": "", //docker数据持久化存储的根目录
"dns": [], //DNS服务器
"dns-opts": [], //DNS配置选项，如端口等
"dns-search": [], //DNS搜索域名
"exec-opts": [], //执行选项
"exec-root": "", //执行状态的文件的根目录
"experimental": false, //是否开启试验性特性
"storage-driver": "", //存储驱动器
"storage-opts": [], //存储选项
"labels": [], //键值对式标记docker元数据
"live-restore": true, //dockerd挂掉是否保活容器（避免了docker服务异常而造成容器退出）
"log-driver": "", //容器日志的驱动器
"log-opts": {}, //容器日志的选项
, //设置容器网络MTU（最大传输单元）
"pidfile": "", //daemon PID文件的位置
"cluster-store": "", //集群存储系统的URL
"cluster-store-opts": {}, //配置集群存储
"cluster-advertise": "", //对外的地址名称
, //设置每个pull进程的最大并发
, //设置每个push进程的最大并发
"default-shm-size": "64M", //设置默认共享内存的大小
, //设置关闭的超时时限(who?)
"debug": true, //开启调试模式
"hosts": [], //监听地址(?)
"log-level": "", //日志级别
"tls": true, //开启传输层安全协议TLS
"tlsverify": true, //开启输层安全协议并验证远程地址
"tlscacert": "", //CA签名文件路径
"tlscert": "", //TLS证书文件路径
"tlskey": "", //TLS密钥文件路径
"swarm-default-advertise-addr": "", //swarm对外地址
"api-cors-header": "", //设置CORS（跨域资源共享-Cross-origin resource sharing）头
"selinux-enabled": false, //开启selinux(用户、进程、应用、文件的强制访问控制)
"userns-remap": "", //给用户命名空间设置 用户/组
"group": "", //docker所在组
"cgroup-parent": "", //设置所有容器的cgroup的父类(?)
"default-ulimits": {}, //设置所有容器的ulimit
"init": false, //容器执行初始化，来转发信号或控制(reap)进程
"init-path": "/usr/libexec/docker-init", //docker-init文件的路径
"ipv6": false, //开启IPV6网络
"iptables": false, //开启防火墙规则
"ip-forward": false, //开启net.ipv4.ip_forward
"ip-masq": false, //开启ip掩蔽(IP封包通过路由器或防火墙时重写源IP地址或目的IP地址的技术)
"userland-proxy": false, //用户空间代理
"userland-proxy-path": "/usr/libexec/docker-proxy", //用户空间代理路径
"ip": "0.0.0.0", //默认IP
"bridge": "", //将容器依附(attach)到桥接网络上的桥标识
"bip": "", //指定桥接ip
"fixed-cidr": "", //(ipv4)子网划分，即限制ip地址分配范围，用以控制容器所属网段实现容器间(同一主机或不同主机间)的网络访问
"fixed-cidr-v6": "", //（ipv6）子网划分
"default-gateway": "", //默认网关
"default-gateway-v6": "", //默认ipv6网关
"icc": false, //容器间通信
"raw-logs": false, //原始日志(无颜色、全时间戳)
"allow-nondistributable-artifacts": [], //不对外分发的产品提交的registry仓库
"registry-mirrors": [], //registry仓库镜像
"seccomp-profile": "", //seccomp配置文件
"insecure-registries": [], //非https的registry地址
"no-new-privileges": false, //禁止新优先级(??)
"default-runtime": "runc", //OCI联盟(The Open Container Initiative)默认运行时环境
, //内存溢出被杀死的优先级(-1000~1000)
"node-generic-resources": ["NVIDIA-GPU=UUID1", "NVIDIA-GPU=UUID2"], //对外公布的资源节点
"runtimes": { //运行时
"cc-runtime": {
"path": "/usr/bin/cc-runtime"
},
"custom": {
"path": "/usr/local/bin/my-runc-replacement",
"runtimeArgs": [
"--debug"
]
}
}
}*如果不是墙外的机器,那么还需要该校Docker的代理设置,因为平常使用的终端代理命令对Docker是无效的,需要单独设置,参考文档: Docker使用代理
2.使用kubeadm部署Kubernetes
2.1 安装kubeadm和kubelet
下面在各节点安装kubeadm和kubelet：
cat <<EOF > /etc/yum.repos.d/kubernetes.repo [kubernetes] name=Kubernetes baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64 enabled=1 gpgcheck=1 repo_gpgcheck=1 gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg EOF yum install -y kubelet-1.15.7 kubeadm-1.15.7 kubectl-1.15.7
========================================================================================================================================================================================================================================================================================================================================================================= 正在安装: kubeadm x86_64 1.15.7-0 kubernetes 8.9 M kubectl x86_64 1.15.7-0 kubernetes 9.5 M kubelet x86_64 1.15.7-0 kubernetes 22 M 为依赖而安装: kubernetes-cni x86_64 0.7.5-0 kubernetes 10 M 事务概要 ========================================================================================================================================================================================================================================================================================================================================================================= 安装 3 软件包 (+1 依赖软件包)
如果没有关闭swap,可以修改kubelet的配置去掉这个限制
vi /etc/sysconfig/kubelet #添加 KUBELET_EXTRA_ARGS=--fail-swap-on=false
2.2 使用kubeadm init初始化集群
在各节点开机启动kubelet服务：
systemctl enable kubelet.service
使用 kubeadm config print init-defaults 可以打印集群初始化默认的使用的配置：
[root@k8s-master k8s]# kubeadm config print init-defaults
apiVersion: kubeadm.k8s.io/v1beta2
bootstrapTokens:
- groups:
- system:bootstrappers:kubeadm:default-node-token
token: abcdef.0123456789abcdef
ttl: 24h0m0s
usages:
- signing
- authentication
kind: InitConfiguration
localAPIEndpoint:
advertiseAddress: 1.2.3.4
bindPort: 6443
nodeRegistration:
criSocket: /var/run/dockershim.sock
name: k8s-master
taints:
- effect: NoSchedule
key: node-role.kubernetes.io/master
---
apiServer:
timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns:
type: CoreDNS
etcd:
local:
dataDir: /var/lib/etcd
imageRepository: k8s.gcr.io
kind: ClusterConfiguration
kubernetesVersion: v1.15.0
networking:
dnsDomain: cluster.local
serviceSubnet: 10.96.0.0/12
scheduler: {} * imageRepository 参数是可以从指定的地方来拉取k8s所需镜像的.
制出本次使用kubeadm初始化集群所需的配置文件kubeadm.yam
apiVersion: kubeadm.k8s.io/v1beta2 kind: InitConfiguration localAPIEndpoint: #master master节点的地址 advertiseAddress: 192.168.123.21 bindPort: 6443 nodeRegistration: taints: - effect: PreferNoSchedule key: node-role.kubernetes.io/master --- apiVersion: kubeadm.k8s.io/v1beta2 kind: ClusterConfiguration #改成和yum 安装的软件版本一致 kubernetesVersion: v1.15.7 networking: podSubnet: 10.244.0.0/16
使用kubeadm默认配置初始化的集群，会在master节点打上node-role.kubernetes.io/master:NoSchedule的污点，阻止master节点接受调度运行工作负载。如果想让master节点接受负载可以改成: effect: PreferNoSchedul
在开始初始化集群之前可以使用kubeadm config images pull预先在各个节点上拉取所k8s需要的docker镜像。
接下来使用kubeadm初始化集群 在master 上执行下面的命令：
[root@centos7 k8s]# kubeadm init --config kubeadm.yaml [init] Using Kubernetes version: v1.15.7 [preflight] Running pre-flight checks [preflight] Pulling images required for setting up a Kubernetes cluster [preflight] This might take a minute or two, depending on the speed of your internet connection [preflight] You can also perform this action in beforehand using 'kubeadm config images pull' [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Activating the kubelet service [certs] Using certificateDir folder "/etc/kubernetes/pki" [certs] Generating "ca" certificate and key [certs] Generating "apiserver" certificate and key [certs] apiserver serving cert is signed for DNS names [centos7 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.123.23] [certs] Generating "apiserver-kubelet-client" certificate and key [certs] Generating "front-proxy-ca" certificate and key [certs] Generating "front-proxy-client" certificate and key [certs] Generating "etcd/ca" certificate and key [certs] Generating "etcd/server" certificate and key [certs] etcd/server serving cert is signed for DNS names [centos7 localhost] and IPs [192.168.123.23 127.0.0.1 ::1] [certs] Generating "etcd/healthcheck-client" certificate and key [certs] Generating "etcd/peer" certificate and key [certs] etcd/peer serving cert is signed for DNS names [centos7 localhost] and IPs [192.168.123.23 127.0.0.1 ::1] [certs] Generating "apiserver-etcd-client" certificate and key [certs] Generating "sa" key and public key [kubeconfig] Using kubeconfig folder "/etc/kubernetes" [kubeconfig] Writing "admin.conf" kubeconfig file [kubeconfig] Writing "kubelet.conf" kubeconfig file [kubeconfig] Writing "controller-manager.conf" kubeconfig file [kubeconfig] Writing "scheduler.conf" kubeconfig file [control-plane] Using manifest folder "/etc/kubernetes/manifests" [control-plane] Creating static Pod manifest for "kube-apiserver" [control-plane] Creating static Pod manifest for "kube-controller-manager" [control-plane] Creating static Pod manifest for "kube-scheduler" [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s [apiclient] All control plane components are healthy after 36.502867 seconds [upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace [kubelet] Creating a ConfigMap "kubelet-config-1.15" in namespace kube-system with the configuration for the kubelets in the cluster [upload-certs] Skipping phase. Please see --upload-certs [mark-control-plane] Marking the node centos7 as control-plane by adding the label "node-role.kubernetes.io/master=''" [mark-control-plane] Marking the node centos7 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule] [bootstrap-token] Using token: htipfy.mzl1ypdqfjeiut8g [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles [bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials [bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token [bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster [bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace [addons] Applied essential addon: CoreDNS [addons] Applied essential addon: kube-proxy Your Kubernetes control-plane has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ Then you can join any number of worker nodes by running the following on each as root: kubeadm join 192.168.123.21:6443 --token htipfy.mzl1ypdqfjeiut8g \ --discovery-token-ca-cert-hash sha256:d9f9bd831c438f7eba1397befb5f2896926a7c386ce46d5290eb7207f220de7b
上面记录了完成的初始化输出的内容，根据输出的内容基本上可以看出手动初始化安装一个Kubernetes集群所需要的关键步骤。 其中有以下关键内容：
[kubelet-start] 生成kubelet的配置文件”/var/lib/kubelet/config.yaml”
[certs]生成相关的各种证书
[kubeconfig]生成相关的kubeconfig文件
[control-plane]使用/etc/kubernetes/manifests目录中的yaml文件创建apiserver、controller-manager、scheduler的静态pod
[bootstraptoken]生成token记录下来，后边使用kubeadm join往集群中添加节点时会用到
下面的命令是配置常规用户如何使用kubectl访问集群：
mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config
最后给出了将节点加入集群的命令:
kubeadm join 192.168.123.21:6443 --token htipfy.mzl1ypdqfjeiut8g \ --discovery-token-ca-cert-hash sha256:d9f9bd831c438f7eba1397befb5f2896926a7c386ce46d5290eb7207f220de7b
先执行:
mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config
查看一下集群状态，确认个组件都处于healthy状态：
[root@k8s-master k8s]# kubectl get cs
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Healthy ok
etcd-0 Healthy {"health":"true"}集群初始化如果遇到问题，可以使用下面的命令进行清理：
kubeadm reset ifconfig cni0 down ip link delete cni0 ifconfig flannel.1 down ip link delete flannel.1rm -rf /var/lib/cni/
2.3 安装Pod Network
接下来安装flannel network add-on：
mkdir -p ~/k8s/ && cd ~/k8s curl -O https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml kubectl apply -f kube-flannel.yml
这里注意kube-flannel.yml这个文件里的flannel的镜像是0.11.0，quay.io/coreos/flannel:v0.11.0-amd64
如果Node有多个网卡的话，参考flannel issues 39701，目前需要在kube-flannel.yml中使用–iface参数指定集群主机内网网卡的名称，否则可能会出现dns无法解析。需要将kube-flannel.yml下载到本地，flanneld启动参数加上–iface=<iface-name>
使用kubectl get pod –all-namespaces -o wide确保所有的Pod都处于Running状态。
[root@k8s-master k8s]# kubectl get pod -n kube-system NAME READY STATUS RESTARTS AGE coredns-5d4dd4b4db-md2nx 1/1 Running 0 16h coredns-5d4dd4b4db-zjbbz 1/1 Running 0 16h etcd-k8s-master 1/1 Running 0 16h kube-apiserver-k8s-master 1/1 Running 0 16h kube-controller-manager-k8s-master 1/1 Running 0 16h kube-flannel-ds-amd64-8bwtl 1/1 Running 0 6m kube-flannel-ds-amd64-dff6s 1/1 Running 0 4h22m kube-flannel-ds-amd64-h87mv 1/1 Running 0 17m kube-proxy-4wjx2 1/1 Running 0 17m tiller-deploy-7bf78cdbf7-82j8w 1/1 Running 0 172m
2.4 测试集群DNS是否可用
kubectl run curl --image=radial/busyboxplus:curl -it kubectl run --generator=deployment/apps.v1beta1 is DEPRECATED and will be removed in a future version. Use kubectl create instead.If you don't see a command prompt, try pressing enter. [ root@curl-5cc7b478b6-r997p:/ ]$
进入后执行nslookup kubernetes.default确认解析正常:
nslookup kubernetes.default Server: 10.96.0.10 Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local Name: kubernetes.default Address 1: 10.96.0.1 kubernetes.default.svc.cluster.local
2.5 向Kubernetes集群中添加Node节点
在k8s-node1 和k8s-node2 机器上执行在k8s-master 上初始化集群成功最后给出的集群加入命令
[root@centos7 k8s]# kubeadm join 192.168.123.21:6443 --token 7janlf.71lp9ynflfhbt6dr \ > --discovery-token-ca-cert-hash sha256:1e1785429bdc74a2f9712cdd7ad4d8c25711a7f9c44acb5ea8e995df91cf96f6 [preflight] Running pre-flight checks [preflight] Reading configuration from the cluster... [preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml' [kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.15" ConfigMap in the kube-system namespace [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Activating the kubelet service [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... This node has joined the cluster: * Certificate signing request was sent to apiserver and a response was received. * The Kubelet was informed of the new secure connection details. Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
在master节点上执行命令查看集群中的节点：
[root@k8s-master k8s]# kubectl get node NAME STATUS ROLES AGE VERSION k8s-master Ready master 16h v1.15.7 k8s-node1 Ready <none> 11m v1.15.7 k8s-node2 Ready <none> 22m v1.15.7
2.5.1 从集群移除Node
此处以移除k8s-node2节点为例子
在k8s-master机器上执行
kubectl drain k8s-node2 --delete-local-data --force --ignore-daemonsets kubectl delete node k8s-node2
在k8s-node2 机器上执行
kubeadm reset ifconfig cni0 down ip link delete cni0 ifconfig flannel.1 down ip link delete flannel.1rm -rf /var/lib/cni/
2.6 kube-proxy开启ipvs
修改ConfigMap的kube-system/kube-proxy中的config.conf，mode: “ipvs”
kubectl edit cm kube-proxy -n kube-system
# Please edit the object below. Lines beginning with a '#' will be ignored, # and an empty file will abort the edit. If an error occurs while saving this file will be # reopened with the relevant failures. # apiVersion: v1 data: config.conf: |- apiVersion: kubeproxy.config.k8s.io/v1alpha1 bindAddress: 0.0.0.0 clientConnection: acceptContentTypes: "" burst: 10 contentType: application/vnd.kubernetes.protobuf kubeconfig: /var/lib/kube-proxy/kubeconfig.conf qps: 5 clusterCIDR: 10.244.0.0/16 configSyncPeriod: 15m0s conntrack: maxPerCore: 32768 min: 131072 tcpCloseWaitTimeout: 1h0m0s tcpEstablishedTimeout: 24h0m0s enableProfiling: false healthzBindAddress: 0.0.0.0:10256 hostnameOverride: "" iptables: masqueradeAll: false masqueradeBit: 14 minSyncPeriod: 0s syncPeriod: 30s ipvs: excludeCIDRs: null minSyncPeriod: 0s scheduler: "" strictARP: false syncPeriod: 30s kind: KubeProxyConfiguration metricsBindAddress: 127.0.0.1:10249 mode: "ipvs" # 修改此处 nodePortAddresses: null oomScoreAdj: -999 portRange: "" resourceContainer: /kube-proxy udpIdleTimeout: 250ms winkernel: …… …… ……
之后重启各个节点上的kube-proxy pod：
kubectl get pod -n kube-system | grep kube-proxy | awk '{system("kubectl delete pod "$1" -n kube-system")}' 验证是否修改成功
[root@k8s-master k8s]# kubectl get pod -n kube-system | grep kube-proxy kube-proxy-7dr7f 1/1 Running 0 18m kube-proxy-g85qp 1/1 Running 0 3m17s kube-proxy-gqpbg 1/1 Running 0 4h29m [root@k8s-master k8s]# kubectl logs kube-proxy-7dr7f -n kube-system I1218 13:03:35.653032 1 server_others.go:170] Using ipvs Proxier.**** 显示是ipvs了 W1218 13:03:35.653278 1 proxier.go:401] IPVS scheduler not specified, use rr by default I1218 13:03:35.654608 1 server.go:534] Version: v1.15.7 I1218 13:03:35.667966 1 conntrack.go:52] Setting nf_conntrack_max to 131072 I1218 13:03:35.668507 1 config.go:187] Starting service config controller I1218 13:03:35.668518 1 controller_utils.go:1029] Waiting for caches to sync for service config controller I1218 13:03:35.668532 1 config.go:96] Starting endpoints config controller I1218 13:03:35.668537 1 controller_utils.go:1029] Waiting for caches to sync for endpoints config controller I1218 13:03:35.769221 1 controller_utils.go:1036] Caches are synced for endpoints config controller I1218 13:03:35.769255 1 controller_utils.go:1036] Caches are synced for service config controller
日志中打印出了Using ipvs Proxier，说明ipvs模式已经开启。
3.Kubernetes常用组件部署
越来越多的公司和团队开始使用Helm这个Kubernetes的包管理器，这里也将使用Helm安装Kubernetes的常用组件。
3.1 Helm的安装
Helm由客户端命helm令行工具和服务端tiller组成，Helm的安装十分简单。 下载helm命令行工具到master节点node1的/usr/local/bin下，这里下载的2.14.1版本：
curl -O https://get.helm.sh/helm-v2.14.1-linux-amd64.tar.gz tar -zxvf helm-v2.14.1-linux-amd64.tar.gz cd linux-amd64/ cp helm /usr/local/bin/
为了安装服务端tiller，还需要在这台机器上配置好kubectl工具和kubeconfig文件，确保kubectl工具可以在这台机器上访问apiserver且正常使用。 这里的node1节点已经配置好了kubectl。
因为Kubernetes APIServer开启了RBAC访问控制，所以需要创建tiller使用的service account: tiller并分配合适的角色给它。 详细内容可以查看helm文档中的Role-based Access Control。 这里简单起见直接分配cluster-admin这个集群内置的ClusterRole给它。
创建helm-rbac.yaml文件：
apiVersion: v1 kind: ServiceAccount metadata: name: tiller namespace: kube-system --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRoleBinding metadata: name: tiller roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: cluster-admin subjects: - kind: ServiceAccount name: tiller namespace: kube-system
文件创建完执行
kubectl create -f helm-rbac.yaml
接下来使用helm部署tiller:
helm init --service-account tiller --skip-refresh
tiller默认被部署在k8s集群中的kube-system这个namespace下：
[root@k8s-master k8s]# kubectl get pod -n kube-system -l app=helm
NAME READY STATUS RESTARTS AGE
tiller-deploy-7bf78cdbf7-82j8w 1/1 Running 0 3h13m
[root@k8s-master k8s]# helm version
Client: &version.Version{SemVer:"v2.14.1", GitCommit:"5270352a09c7e8b6e8c9593002a73535276507c0", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.14.1", GitCommit:"5270352a09c7e8b6e8c9593002a73535276507c0", GitTreeState:"clean"}注意由于某些原因需要网络可以访问gcr.io和kubernetes-charts.storage.googleapis.com，如果无法访问可以通过helm init –service-account tiller –tiller-image <your-docker-registry>/tiller:v2.13.1 –skip-refresh使用私有镜像仓库中的tiller镜像
最后在k8s-master 上修改helm chart仓库的地址为azure提供的镜像地址：
[root@k8s-master k8s]# helm repo add stable http://mirror.azure.cn/kubernetes/charts "stable" has been added to your repositories [root@k8s-master k8s]# helm repo list NAME URL stable http://mirror.azure.cn/kubernetes/charts local http://127.0.0.1:8879/charts
3.2 使用Helm部署Nginx Ingress
为了便于将集群中的服务暴露到集群外部，需要使用Ingress。接下来使用Helm将Nginx Ingress部署到Kubernetes上。 Nginx Ingress Controller被部署在Kubernetes的边缘节点上，关于Kubernetes边缘节点的高可用相关的内容可以查看之前整理的Bare metal环境下Kubernetes Ingress边缘节点的高可用,Ingress Controller使用hostNetwork。
我们将k8s-master 做为边缘节点，打上Label：
[root@k8s-master k8s]# kubectl label node k8s-master node-role.kubernetes.io/edge= node/k8s-master labeled [root@k8s-master k8s]# kubectl get nodes NAME STATUS ROLES AGE VERSION k8s-master Ready edge,master 17h v1.15.7 k8s-node1 Ready <none> 23m v1.15.7 k8s-node2 Ready <none> 8m5s v1.15.7
stable/nginx-ingress chart的值文件ingress-nginx.yaml如下：
controller: replicaCount: 1 hostNetwork: true nodeSelector: node-role.kubernetes.io/edge: '' affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: app operator: In values: - nginx-ingress - key: component operator: In values: - controller topologyKey: kubernetes.io/hostname tolerations: - key: node-role.kubernetes.io/master operator: Exists effect: NoSchedule - key: node-role.kubernetes.io/master operator: Exists effect: PreferNoSchedule defaultBackend: nodeSelector: node-role.kubernetes.io/edge: '' tolerations: - key: node-role.kubernetes.io/master operator: Exists effect: NoSchedule - key: node-role.kubernetes.io/master operator: Exists effect: PreferNoSchedule
nginx ingress controller的副本数replicaCount为1，将被调度到node1这个边缘节点上。这里并没有指定nginx ingress controller service的externalIPs，而是通过hostNetwork: true设置nginx ingress controller使用宿主机网络。
helm repo update helm install stable/nginx-ingress \ -n nginx-ingress \ --namespace ingress-nginx \ -f ingress-nginx.yaml
[root@k8s-master k8s]# kubectl get pod -n ingress-nginx -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES nginx-ingress-controller-8f9ff8fbf-dr4p4 1/1 Running 0 64s 192.168.123.21 k8s-master <none> <none> nginx-ingress-default-backend-6fdf85fc65-bszxt 1/1 Running 0 64s 10.244.0.6 k8s-master <none> <none>
如果访问http://192.168.123.21 返回default backend，则部署完成。
3.3 使用Helm部署dashboard
创建 kubernetes-dashboard.yaml
image: repository: k8s.gcr.io/kubernetes-dashboard-amd64 tag: v1.10.1 ingress: enabled: true hosts: - k8s.frognew.com #替换域名 annotations: nginx.ingress.kubernetes.io/ssl-redirect: "true" nginx.ingress.kubernetes.io/backend-protocol: "HTTPS" tls: - secretName: frognew-com-tls-secret hosts: - k8s.frognew.com #替换域名 nodeSelector: node-role.kubernetes.io/edge: '' tolerations: - key: node-role.kubernetes.io/master operator: Exists effect: NoSchedule - key: node-role.kubernetes.io/master operator: Exists effect: PreferNoSchedule rbac: clusterAdminRole: true
我是从路由器的dns 中加了一条 192.168.123.21 k8s.loacl.com 的记录来实现本地局域网访问的.
helm install stable/kubernetes-dashboard \ -n kubernetes-dashboard \ --namespace kube-system \ -f kubernetes-dashboard.yaml
查看登录的token
[root@k8s-master k8s]# kubectl -n kube-system get secret | grep kubernetes-dashboard-token kubernetes-dashboard-token-ss5hv kubernetes.io/service-account-token 3 3m15s [root@k8s-master k8s]# kubectl describe -n kube-system secret/kubernetes-dashboard-token-ss5hv Name: kubernetes-dashboard-token-ss5hv Namespace: kube-system Labels: <none> Annotations: kubernetes.io/service-account.name: kubernetes-dashboard kubernetes.io/service-account.uid: 589585a2-0024-4e77-965d-8dcf22a7b1a3 Type: kubernetes.io/service-account-token Data ==== ca.crt: 1025 bytes namespace: 11 bytes token: eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJrdWJlcm5ldGVzLWRhc2hib2FyZC10b2tlbi1zczVodiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjU4OTU4NWEyLTAwMjQtNGU3Ny05NjVkLThkY2YyMmE3YjFhMyIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlLXN5c3RlbTprdWJlcm5ldGVzLWRhc2hib2FyZCJ9.Xy2wDRpB4_fTAEncq3I4X9xxRpioLlZ4zUYaiSpeIyvTVahTmSxK23GlI9AxquQRguiy_MXCo9nmshmz6bjhLuXRbEcuwoeV-RaLdyt_k1o2ko7xv0ftBwLC0ZeLeto-JD9zabd-7rZMeo2Vdmrd6UIXSBQay5r-njuGRtS_VDh4w4YIZVrnoWdGS5fcJzUya-glNSzHNF3ka0H2-c3ZlbU8AkdRgQaq68As5rsovg76UvlkG1oK0g7o9Zg1Az6ELeWLm3z0nMGWLORBnxH3fZgPNXCNWJkvTrD9kvyrUTZCgkCCkE7C6ViWMr8USHJCcv2u7Z3NrH8Nw_WsVudjMA
在机器上输入https://k8s.gyx.com 就可以看到dashboard 的登录界面了
3.4 使用Helm部署metrics-server
从Heapster的github https://github.com/kubernetes/heapster中可以看到已经，heapster已经DEPRECATED。 这里是heapster的deprecation timeline。 可以看出heapster从Kubernetes 1.12开始从Kubernetes各种安装脚本中移除。
Kubernetes推荐使用metrics-server。我们这里也使用helm来部署metrics-server。
创建 metrics-server.yaml
args: - --logtostderr - --kubelet-insecure-tls - --kubelet-preferred-address-types=InternalIP nodeSelector: node-role.kubernetes.io/edge: '' tolerations: - key: node-role.kubernetes.io/master operator: Exists effect: NoSchedule - key: node-role.kubernetes.io/master operator: Exists effect: PreferNoSchedule
helm install stable/metrics-server \ -n metrics-server \ --namespace kube-system \ -f metrics-server.yaml
使用下面的命令可以获取到关于集群节点基本的指标信息：
[root@k8s-master k8s]# kubectl top node NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% k8s-master 144m 7% 2452Mi 63% k8s-node1 35m 1% 1418Mi 36% k8s-node2 49m 2% 1415Mi 36% [root@k8s-master k8s]# kubectl top pod -n kube-system NAME CPU(cores) MEMORY(bytes) coredns-5d4dd4b4db-md2nx 4m 19Mi coredns-5d4dd4b4db-zjbbz 3m 13Mi etcd-k8s-master 18m 72Mi kube-apiserver-k8s-master 32m 263Mi kube-controller-manager-k8s-master 19m 73Mi kube-flannel-ds-amd64-6z8xd 4m 12Mi kube-flannel-ds-amd64-8bwtl 2m 14Mi kube-flannel-ds-amd64-dff6s 2m 12Mi kube-proxy-7dr7f 1m 16Mi kube-proxy-g85qp 1m 18Mi kube-proxy-gqpbg 3m 24Mi kube-scheduler-k8s-master 2m 23Mi kubernetes-dashboard-848b8dd798-8dfbn 1m 13Mi metrics-server-ff86b4bc5-rg6fs 2m 12Mi tiller-deploy-7bf78cdbf7-82j8w 1m 34Mi
遗憾的是，当前Kubernetes Dashboard还不支持metrics-server。因此如果使用metrics-server替代了heapster，将无法在dashboard中以图形展示Pod的内存和CPU情况(实际上这也不是很重要，当前我们是在Prometheus和Grafana中定制的Kubernetes集群中各个Pod的监控，因此在dashboard中查看Pod内存和CPU也不是很重要)。 Dashboard的github上有很多这方面的讨论，如https://github.com/kubernetes/dashboard/issues/2986，Dashboard已经准备在将来的某个时间点支持metrics-server。但由于metrics-server和metrics pipeline肯定是Kubernetes在monitor方面未来的方向，所以推荐使用metrics-server。
参考文档:
还没有评论，来说两句吧...