轻课堂首页

播放记录

Kubernetes学习笔记-手动搭建k8s-1.10.4之部署flannel网络

博客 > 术业专攻> 云计算> kubernetes> Kubernetes学习笔记-手动搭建k8s-1.10.4之部署flannel网络 2019年08月29日 11:24:38

kubernetes 要求集群内各节点(包括 master 节点)能通过 Pod 网段互联互通。flannel 使用 vxlan 技术为各节点创建一个可以互通的 Pod 网络，使用的端口为 UDP 8472，需要开放该端口（如公有云 AWS 等）。

flannel 第一次启动时，从 etcd 获取 Pod 网段信息，为本节点分配一个未使用的 /24 段地址，然后创建 flannel.1（也可能是其它名称，如 flannel1 等） 接口。

flannel 将分配的 Pod 网段信息写入 /run/flannel/docker 文件，docker 后续使用这个文件中的环境变量设置 docker0 网桥。

1，下载和分发 flanneld 二进制文件

到 https://github.com/coreos/flannel/releases 页面下载最新版本的发布包：

mkdir flannel
wget https://github.com/coreos/flannel/releases/download/v0.10.0/flannel-v0.10.0-linux-amd64.tar.gz
tar -xzvf flannel-v0.10.0-linux-amd64.tar.gz -C flannel

分发 flanneld 二进制文件到集群所有节点：

cat > magic.sh << "EOF"
#!/bin/bash

source /opt/k8s/bin/environment.sh

for node_ip in ${NODE_IPS[@]}
do
    echo ">>> ${node_ip}" 
    scp  flannel/{flanneld,mk-docker-opts.sh} k8s@${node_ip}:/opt/k8s/bin/
    ssh k8s@${node_ip} "chmod +x /opt/k8s/bin/*"
done
EOF

2，创建 flannel 证书和私钥

flannel 从 etcd 集群存取网段分配信息，而 etcd 集群启用了双向 x509 证书认证，所以需要为 flanneld 生成证书和私钥。

创建证书签名请求：

cat > flanneld-csr.json <
  • 该证书只会被 kubectl 当做 client 证书使用，所以 hosts 字段为空；

生成证书和私钥：

$cfssl gencert -ca=/etc/kubernetes/cert/ca.pem \
  -ca-key=/etc/kubernetes/cert/ca-key.pem \
  -config=/etc/kubernetes/cert/ca-config.json \
  -profile=kubernetes flanneld-csr.json | cfssljson -bare flanneld

$ls flanneld*pem

将生成的证书和私钥分发到所有节点（master 和 worker）：

cat > magic.sh << "EOF"
#!/bin/bash

source /opt/k8s/bin/environment.sh

for node_ip in ${NODE_IPS[@]}
do
    echo ">>> ${node_ip}" 
    ssh root@${node_ip} "mkdir -p /etc/flanneld/cert && chown -R k8s /etc/flanneld"
    scp flanneld*.pem k8s@${node_ip}:/etc/flanneld/cert
done
EOF

3，向 etcd 写入集群 Pod 网段信息

注意：本步骤只需执行一次。

$source /opt/k8s/bin/environment.sh

$etcdctl \
  --endpoints=${ETCD_ENDPOINTS} \
  --ca-file=/etc/kubernetes/cert/ca.pem \
  --cert-file=/etc/flanneld/cert/flanneld.pem \
  --key-file=/etc/flanneld/cert/flanneld-key.pem \
  set ${FLANNEL_ETCD_PREFIX}/config '{"Network":"'${CLUSTER_CIDR}'", "SubnetLen": 24, "Backend": {"Type": "vxlan"}}'
  • flanneld 当前版本 (v0.10.0) 不支持 etcd v3，故使用 etcd v2 API 写入配置 key 和网段数据；
  • 写入的 Pod 网段 ${CLUSTER_CIDR} 必须是 /16 段地址，必须与 kube-controller-manager 的 --cluster-cidr 参数值一致；

4，创建 flanneld 的 systemd unit 文件

$source /opt/k8s/bin/environment.sh

$cat > flanneld.service << EOF
[Unit]
Description=Flanneld overlay address etcd agent
After=network.target
After=network-online.target
Wants=network-online.target
After=etcd.service
Before=docker.service

[Service]
Type=notify
ExecStart=/opt/k8s/bin/flanneld \\
  -etcd-cafile=/etc/kubernetes/cert/ca.pem \\
  -etcd-certfile=/etc/flanneld/cert/flanneld.pem \\
  -etcd-keyfile=/etc/flanneld/cert/flanneld-key.pem \\
  -etcd-endpoints=${ETCD_ENDPOINTS} \\
  -etcd-prefix=${FLANNEL_ETCD_PREFIX} \\
  -iface=${VIP_IF}
ExecStartPost=/opt/k8s/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker
Restart=on-failure

[Install]
WantedBy=multi-user.target
RequiredBy=docker.service
EOF
  • mk-docker-opts.sh 脚本将分配给 flanneld 的 Pod 子网网段信息写入 /run/flannel/docker 文件，后续 docker 启动时使用这个文件中的环境变量配置 docker0 网桥；
  • flanneld 使用系统缺省路由所在的接口与其它节点通信，对于有多个网络接口（如内网和公网）的节点，可以用 -iface 参数指定通信接口，如上面的 eth0 接口;
  • flanneld 运行时需要 root 权限；

5，分发 flanneld systemd unit 文件到所有节点

cat > magic.sh << "EOF"
#!/bin/bash

source /opt/k8s/bin/environment.sh

for node_ip in ${NODE_IPS[@]}
do
    echo ">>> ${node_ip}" 
    scp flanneld.service root@${node_ip}:/etc/systemd/system/
done
EOF

6，启动 flanneld 服务

cat > magic.sh << "EOF"
#!/bin/bash

source /opt/k8s/bin/environment.sh

for node_ip in ${NODE_IPS[@]}
do
    echo ">>> ${node_ip}" 
    ssh root@${node_ip} "systemctl daemon-reload && systemctl enable flanneld && systemctl start flanneld"
done
EOF

7，检查启动结果

cat > magic.sh << "EOF"
#!/bin/bash

source /opt/k8s/bin/environment.sh

for node_ip in ${NODE_IPS[@]}
do
    echo ">>> ${node_ip}" 
    ssh k8s@${node_ip} "systemctl status flanneld|grep Active"
done
EOF

看到如下输出：

$bash magic.sh
>>> 192.168.106.3
   Active: active (running) since Fri 2018-11-23 17:11:40 CST; 6h ago
>>> 192.168.106.4
   Active: active (running) since Fri 2018-11-23 17:11:40 CST; 6h ago
>>> 192.168.106.5
   Active: active (running) since Fri 2018-11-23 17:11:41 CST; 6h ago

则说明正常，如果失败，则用如下命令查看日志：

journalctl -ux flanneld

8，检查分配给各 flanneld 的 Pod 网段信息

查看集群 Pod 网段(/16)：

source /opt/k8s/bin/environment.sh

etcdctl \
  --endpoints=${ETCD_ENDPOINTS} \
  --ca-file=/etc/kubernetes/cert/ca.pem \
  --cert-file=/etc/flanneld/cert/flanneld.pem \
  --key-file=/etc/flanneld/cert/flanneld-key.pem \
  get ${FLANNEL_ETCD_PREFIX}/config

输出：

{"Network":"172.30.0.0/16","SubnetLen": 24, "Backend": {"Type": "vxlan"}}

查看已分配的 Pod 子网段列表(/24):

source /opt/k8s/bin/environment.sh

etcdctl \
  --endpoints=${ETCD_ENDPOINTS} \
  --ca-file=/etc/kubernetes/cert/ca.pem \
  --cert-file=/etc/flanneld/cert/flanneld.pem \
  --key-file=/etc/flanneld/cert/flanneld-key.pem \
  ls ${FLANNEL_ETCD_PREFIX}/subnets

输出：

/kubernetes/network/subnets/172.30.84.0-24
/kubernetes/network/subnets/172.30.8.0-24
/kubernetes/network/subnets/172.30.29.0-24

查看某一 Pod 网段对应的节点 IP 和 flannel 接口地址:

注意其中的IP段换成自己的。

source /opt/k8s/bin/environment.sh

etcdctl \
  --endpoints=${ETCD_ENDPOINTS} \
  --ca-file=/etc/kubernetes/cert/ca.pem \
  --cert-file=/etc/flanneld/cert/flanneld.pem \
  --key-file=/etc/flanneld/cert/flanneld-key.pem \
  get ${FLANNEL_ETCD_PREFIX}/subnets/172.30.8.0-24

输出：

{"PublicIP":"192.168.106.4","BackendType":"vxlan","BackendData":{"VtepMAC":"f2:14:20:50:4f:af"}}

9，验证各节点能通过 Pod 网段互通

在各节点上部署 flannel 后，检查是否创建了 flannel 接口(名称可能为 flannel0、flannel.0、flannel.1 等)：

cat > magic.sh << "EOF"
#!/bin/bash

source /opt/k8s/bin/environment.sh

for node_ip in ${NODE_IPS[@]}
do
    echo ">>> ${node_ip}" 
    ssh ${node_ip} "/usr/sbin/ip addr show flannel.1|grep -w inet"
done
EOF

输出：

$bash magic.sh
>>> 192.168.106.3
    inet 172.30.84.0/32 scope global flannel.1
>>> 192.168.106.4
    inet 172.30.8.0/32 scope global flannel.1
>>> 192.168.106.5
    inet 172.30.29.0/32 scope global flannel.1

在各节点上 ping 所有 flannel 接口 IP，确保能通：

注意其中的IP段换成自己的。

cat > magic.sh << "EOF"
#!/bin/bash

source /opt/k8s/bin/environment.sh

for node_ip in ${NODE_IPS[@]}
do
    echo ">>> ${node_ip}" 
    ssh ${node_ip} "ping -c 1 172.30.8.0"
    ssh ${node_ip} "ping -c 1 172.30.29.0"
    ssh ${node_ip} "ping -c 1 172.30.84.0"
done
EOF

输出：

$bash magic.sh
>>> 192.168.106.3
PING 172.30.8.0 (172.30.8.0) 56(84) bytes of data.
64 bytes from 172.30.8.0: icmp_seq=1 ttl=64 time=0.285 ms

--- 172.30.8.0 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.285/0.285/0.285/0.000 ms
PING 172.30.29.0 (172.30.29.0) 56(84) bytes of data.
64 bytes from 172.30.29.0: icmp_seq=1 ttl=64 time=0.337 ms

--- 172.30.29.0 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.337/0.337/0.337/0.000 ms
PING 172.30.84.0 (172.30.84.0) 56(84) bytes of data.
64 bytes from 172.30.84.0: icmp_seq=1 ttl=64 time=0.062 ms

--- 172.30.84.0 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.062/0.062/0.062/0.000 ms
>>> 192.168.106.4
PING 172.30.8.0 (172.30.8.0) 56(84) bytes of data.
64 bytes from 172.30.8.0: icmp_seq=1 ttl=64 time=0.055 ms

--- 172.30.8.0 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.055/0.055/0.055/0.000 ms
PING 172.30.29.0 (172.30.29.0) 56(84) bytes of data.
64 bytes from 172.30.29.0: icmp_seq=1 ttl=64 time=0.311 ms

--- 172.30.29.0 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.311/0.311/0.311/0.000 ms
PING 172.30.84.0 (172.30.84.0) 56(84) bytes of data.
64 bytes from 172.30.84.0: icmp_seq=1 ttl=64 time=0.395 ms

--- 172.30.84.0 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.395/0.395/0.395/0.000 ms
>>> 192.168.106.5
PING 172.30.8.0 (172.30.8.0) 56(84) bytes of data.
64 bytes from 172.30.8.0: icmp_seq=1 ttl=64 time=0.325 ms

--- 172.30.8.0 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.325/0.325/0.325/0.000 ms
PING 172.30.29.0 (172.30.29.0) 56(84) bytes of data.
64 bytes from 172.30.29.0: icmp_seq=1 ttl=64 time=0.060 ms

--- 172.30.29.0 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.060/0.060/0.060/0.000 ms
PING 172.30.84.0 (172.30.84.0) 56(84) bytes of data.
64 bytes from 172.30.84.0: icmp_seq=1 ttl=64 time=0.260 ms

--- 172.30.84.0 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.260/0.260/0.260/0.000 ms
转载：http://www.eryajf.net/2193.html
互联网 游戏分享 建站/维护 思科网络 SEO SEO问答 SEO算法 SEO诊断案例 吉他谱 其他 计算机技术 软件/应用 网络推广 站内事务 智商税 k8s Jenkins kubernetes 系统优化 docker 迎刃而解 nginx rancher Git nexus elk 消息队列 tomcat 云计算 shell maven kvm zabbix prometheus MySQL 镜像 kibana 进程管理 systemd Mac 阿里云 Windows Android redis pinpoint supervisor sonar rsyslog filebeat 微服务 lnmp vscode golang CentOS7

友情链接:

© 2018 www.qingketang.net 鄂ICP备18027844号-1

武汉快勤科技有限公司 13554402156 武汉市东湖新技术开发区关山二路特一号国际企业中心6幢4层7号
发表评论 X

登录成功
开通VIP

订单金额：

支付金额：

支付方式：