使用kubeadm

环境

节点类型

IP地址

CPU

内存

硬盘

Master

192.168.145.100

1

512MB

20GB

Node1

192.168.145.101

1

512MB

20GB

Node2

192.168.145.102

1

512MB

20GB

所有节点的操作系统为Ubuntu16.04

部署

准备预先环境

  1. 关闭swap；

    swapoff -a

  2. 将/etc/fstab中的关于swap的记录删掉；

  3. 关闭防火墙（Ubuntu中默认没有防火墙，可以不敲）；

    systemctl stop firewalld
systemctl disable firewalld

  4. 禁用selinux（Ubuntu中默认没有开selinux，可以不敲）；

    vim /etc/sysconfig/selinux

安装Docker

  1. 此处参考配置Docker官方配置文档即可；

  2. 为Docker添加镜像加速器以及使用systemd作为cgroupdriver:

    cat > /etc/docker/daemon.json <<EOF
{
    "registry-mirrors": ["https://d8ui43mx.mirror.aliyuncs.com"],
    "live-restore": true,
    "exec-opts": ["native.cgroupdriver=systemd"]
}
EOF

    可以使用docker info命令验证是否配置成功

  3. 使配置生效；

    systemctl daemon-reload
systemctl restart docker

安装kubectl，kubelet，kubeadm

官方文档：https://kubernetes.io/docs/setup/independent/install-kubeadm/

https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/

  1. 安装相关必须的软件；

    apt-get update && apt-get install -y apt-transport-https curl
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -

  2. 使用国内的k8s源安装三个软件：

    cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb http://mirrors.ustc.edu.cn/kubernetes/apt kubernetes-xenial main
EOF
apt update
# 此时会报错，提示无法验证签名 NO PUBKEY XXXXXXXXXX，此时需要输入以下两句命令
gpg --keyserver keyserver.ubuntu.com --recv-keys BA07F4FB
gpg --export --armor BA07F4FB | sudo apt-key add -

    注意替换BA07F4FB为你执行时显示出的NO_PUBKEY的后八位

  3. 下载相关软件：

    apt-get update
apt-get install -y kubelet kubeadm kubectl    # 这种方式默认安装的是最新版本
# apt-cache madison kubeadm kubelet kubectl 查看仓库中的版本
# apt install -y kubelet=1.13.3-00 kubeadm=1.13.3-00 kubectl=1.13.3-00    安装指定版本
apt-mark hold kubelet kubeadm kubectl

配置Master

  1. 初始化Master；

    kubeadm init --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=192.168.145.100 --ignore-preflight-errors=NumCPU

    --pod-network-cidr是指配置节点中的Pod可用IP地址，为内部IP，由于这里使用flannel实现内部网络，因此选用该网段地址作为内部地址；

    --apiserver-advertise-address为Master的IP地址；

    --ignore-preflight-errors是因为此处的Master只有1个CPU，而k8s默认要求最少的CPU数为2，为了让其不报错，添加该参数；

  2. 很遗憾，由于镜像被墙了，所以会有如下错误：

    root@Master:~# kubeadm init --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=192.168.145.100 --ignore-preflight-errors=NumCPU
I0509 21:54:20.580657   12842 version.go:96] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get https://dl.k8s.io/release/stable-1.txt: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
I0509 21:54:20.580706   12842 version.go:97] falling back to the local client version: v1.14.1
[init] Using Kubernetes version: v1.14.1
[preflight] Running pre-flight checks
        [WARNING NumCPU]: the number of available CPUs 1 is less than the required 2
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
error execution phase preflight: [preflight] Some fatal errors occurred:
        [ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-apiserver:v1.14.1: output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
, error: exit status 1
        [ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-controller-manager:v1.14.1: output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
, error: exit status 1
        [ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-scheduler:v1.14.1: output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
, error: exit status 1
        [ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-proxy:v1.14.1: output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
, error: exit status 1
        [ERROR ImagePull]: failed to pull image k8s.gcr.io/pause:3.1: output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
, error: exit status 1
        [ERROR ImagePull]: failed to pull image k8s.gcr.io/etcd:3.3.10: output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
, error: exit status 1
        [ERROR ImagePull]: failed to pull image k8s.gcr.io/coredns:1.3.1: output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
, error: exit status 1
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`

    这些错误都是镜像拉取错误，所以我们需要根据这些信息，去国内的网站上手动pull这些镜像，然后修改其tag。

  3. 从阿里云拉取这些镜像，然后逐个重新打tag：

    docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.14.1
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.14.1
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.14.1
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.14.1
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.3.10
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.3.1

    注意版本号要与上面错误提示中的镜像版本号一致

    docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.14.1 k8s.gcr.io/kube-proxy:v1.14.1
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.14.1 k8s.gcr.io/kube-apiserver:v1.14.1
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.14.1 k8s.gcr.io/kube-controller-manager:v1.14.1
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.14.1 k8s.gcr.io/kube-scheduler:v1.14.1
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.3.1 k8s.gcr.io/coredns:1.3.1
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.3.10 k8s.gcr.io/etcd:3.3.10
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1 k8s.gcr.io/pause:3.1

  4. 重新初始化Master节点：

    kubeadm init --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=192.168.145.100 --ignore-preflight-errors=NumCPU

    会出现以下提示，提示下一步需要配置使常规用户也能使用，配置Pod的网络：

    Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.145.100:6443 --token b96oe5.fkh73ya5of39wv7f \
    --discovery-token-ca-cert-hash sha256:f9bf19abd3b417c17926baec4b079d15947ab6460f44c4216f51194de1d0f40c

  5. 配置使常规用户也能使用（在常规用户下使用以下命令）：

    mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

    同时由于安装是在root下安装的，因此在root下需使用以下命令，使用该命令可以解决报无法访问8080接口的问题：

    echo "export KUBECONFIG=/etc/kubernetes/admin.conf" >> ~/.bashrc
source ~/.bashrc

    为了使用自动补全功能，添加以下语句：

    echo "source <(kubectl completion bash)" >> ~/.bashrc
source ~/.bashrc

  6. 配置Pod网络：

    根据提示，可以使用kubectl apply -f [podnetwork].yaml命令配置网络，网络类型有多种，可以参考该网站：https://kubernetes.io/docs/concepts/cluster-administration/addons/

    这里使用的是flannel格式，因此相关的yaml文件可以从此处查看：https://github.com/coreos/flannel/blob/master/Documentation/kube-flannel.yml

    将该文件保存至/root/目录下，然后通过以下命令配置网络：

    kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/a70459be0084506e4ec919aa1c114638878db11b/Documentation/kube-flannel.yml

# 有如下提示：
podsecuritypolicy.extensions/psp.flannel.unprivileged created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.extensions/kube-flannel-ds-amd64 created
daemonset.extensions/kube-flannel-ds-arm64 created
daemonset.extensions/kube-flannel-ds-arm created
daemonset.extensions/kube-flannel-ds-ppc64le created
daemonset.extensions/kube-flannel-ds-s390x created

  7. （额外的）默认情况下，Master节点是不能够创建Pod的，因此如果是单机环境的话，需要执行以下命令解除限制：

    kubectl taint nodes --all node-role.kubernetes.io/master-

配置Node节点

  1. 加入Node（使用root用户）

    kubeadm join 192.168.145.100:6443 --token b96oe5.fkh73ya5of39wv7f \
    --discovery-token-ca-cert-hash sha256:f9bf19abd3b417c17926baec4b079d15947ab6460f44c4216f51194de1d0f40c

  2. 在Master上检查运行状况：

    root@Master:~# kubectl get nodes
NAME     STATUS     ROLES    AGE     VERSION
master   Ready      master   59m     v1.14.1
node1    NotReady   <none>   3m33s   v1.14.1
node2    NotReady   <none>   3m41s   v1.14.1

root@Master:~# kubectl get pod --all-namespaces
NAMESPACE     NAME                             READY   STATUS              RESTARTS   AGE
kube-system   coredns-fb8b8dccf-9gvs5          1/1     Running             0          45m
kube-system   coredns-fb8b8dccf-nqlwr          1/1     Running             0          45m
kube-system   etcd-master                      1/1     Running             0          58m
kube-system   kube-apiserver-master            1/1     Running             0          57m
kube-system   kube-controller-manager-master   1/1     Running             0          46m
kube-system   kube-flannel-ds-amd64-7dnbh      0/1     Init:0/1            0          2m58s
kube-system   kube-flannel-ds-amd64-dk4t6      0/1     Init:0/1            0          2m51s
kube-system   kube-flannel-ds-amd64-s5wwq      1/1     Running             0          11m
kube-system   kube-proxy-8rhqf                 0/1     ContainerCreating   0          2m51s
kube-system   kube-proxy-9gh84                 1/1     Running             0          45m
kube-system   kube-proxy-pxj9b                 0/1     ContainerCreating   0          2m58s
kube-system   kube-scheduler-master            1/1     Running             0          58m

    我们发现每个Node节点的kube-proxy以及kube-flannel-ds-amd64都没有成功，我们可以使用如下命令去查看该pod的详细信息日志：

    kubectl describe pod kube-proxy-8rhqf --namespace=kube-system
kubectl describe pod kube-flannel-ds-amd64-7dnbh --namespace=kube-system

    在底部我们会发现是因为该节点上没有k8s.gcr.io/pause:3.1、k8s.gcr.io/kube-proxy:v1.14.1和quay.io/coreos/flannel:v0.11.0-amd64这三个镜像，因此我们根据之前在Master上pull镜像的操作，为两个Node节点也pull该镜像即可。

    使用以下命令手动pull并且tag即可：

    docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.14.1
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1
docker pull quay-mirror.qiniu.com/coreos/flannel:v0.11.0-amd64

docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.14.1 k8s.gcr.io/kube-proxy:v1.14.1
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1 k8s.gcr.io/pause:3.1
docker tag quay-mirror.qiniu.com/coreos/flannel:v0.11.0-amd64 quay.io/coreos/flannel:v0.11.0-amd64

  3. 再次确认状态，可以发现都是Ready了：

     root@Master:~# kubectl get nodes
 NAME     STATUS   ROLES    AGE    VERSION
 master   Ready    master   118m   v1.14.1
 node1    Ready    <none>   62m    v1.14.1
 node2    Ready    <none>   62m    v1.14.1

 root@Master:~# kubectl get pods --all-namespaces
 NAMESPACE     NAME                             READY   STATUS    RESTARTS   AGE
 kube-system   coredns-fb8b8dccf-9gvs5          1/1     Running   0          104m
 kube-system   coredns-fb8b8dccf-nqlwr          1/1     Running   0          104m
 kube-system   etcd-master                      1/1     Running   0          117m
 kube-system   kube-apiserver-master            1/1     Running   0          116m
 kube-system   kube-controller-manager-master   1/1     Running   0          105m
 kube-system   kube-flannel-ds-amd64-7dnbh      1/1     Running   0          61m
 kube-system   kube-flannel-ds-amd64-dk4t6      1/1     Running   0          61m
 kube-system   kube-flannel-ds-amd64-s5wwq      1/1     Running   0          70m
 kube-system   kube-proxy-8rhqf                 1/1     Running   0          61m
 kube-system   kube-proxy-9gh84                 1/1     Running   0          104m
 kube-system   kube-proxy-pxj9b                 1/1     Running   0          61m
 kube-system   kube-scheduler-master            1/1     Running   0          117m

清除（Tear down）

清除整个集群

以下操作在Master以及所有的Node上都执行！！！

  1. 执行reset命令：

    kubeadm reset

  2. 清除iptables规则：

    iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X

  3. 重置IPVS表（IPVS=IP Virtual Server，实现了传输层的负载均衡）：

    ipvsadm --clear

清除指定Node

  1. 在Master上执行以下命令从集群中删除Node：

    kubectl drain <node name> --delete-local-data --force --ignore-daemonsets
kubectl delete node <node name>

  2. 在需要被删除的Node中执行reset命令：

    kubeadm reset

  3. 在需要被删除的Node中清除iptables规则：

    iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X

  4. 在需要被删除的Node中重置IPVS表：

    ipvsadm --clear
Previous创建用户Next安装监控插件

Last updated

Was this helpful?