1. 首页
  2. 问答
  3. kubernetes：无法从其他机器访问NodePort

kubernetes：无法从其他机器访问NodePort

2017-10-10 28 views
1

我正在尝试安装kubernetes。 检查在kubernetes上执行的容器时出现问题。 我将服务的类型设置为Nodeport，但我无法从运行容器的节点以外的节点访问。 想要让它可以从其他电脑访问，请告诉我它在哪里不同。 我试过externalIPs和LoadBarancer，但这是不可能的。kubernetes：无法从其他机器访问NodePort

环境

  • 操作系统：Ubuntu的LTS 16.04
  • Kubernetes：1.8
  • 泊坞窗：17.09.0-CE
  • ETCD：3.2.8
  • 法兰绒：0.9.0

网络

  • 物理：10.1.1.0/24
  • 绒布：172.16.0.0/16
  • 泊坞窗：192.168.0.0/16

机

  • 主节点（2nodes）： 10.1.1.24,10.1.1.25
  • 工作节点（2nodes）：10.1.1.26,10.1.1.27

kubectl描述SVC nginx的群集

Name:    nginx-cluster 
Namespace:   default 
Labels:   app=nginx-demo 
Annotations:  <none> 
Selector:   app=nginx-demo 
Type:    ClusterIP 
IP:    172.16.236.159 
Port:    <unset> 8090/TCP 
TargetPort:  80/TCP 
Endpoints:   192.168.24.2:80 
Session Affinity: None 
Events:   <none>

kubectl描述SVC nginx的服务

Name:      nginx-service 
Namespace:    default 
Labels:     app=nginx-demo 
Annotations:    <none> 
Selector:     app=nginx-demo 
Type:      NodePort 
IP:      172.16.199.69 
Port:      <unset> 8090/TCP 
TargetPort:    80/TCP 
NodePort:     <unset> 31659/TCP 
Endpoints:    192.168.24.2:80 
Session Affinity:   None 
External Traffic Policy: Cluster 
Events:     <none>

运行容器工作节点（10.1.1.27）

卷曲10.1.1.27:31659

<!DOCTYPE html> 
<html> 
<head> 
<title>Welcome to nginx!</title> 
...

工人节点（10.1.1.26）

卷曲10.1.1.27:31659

curl: (7) Failed to connect to 10.1.1.27 port 31659:Connection timed out.

其它机器（10.1.1.XX）

卷曲10.1.1。27：31659

curl: (7) Failed to connect to 10.1.1.27 port 31659:Connection timed out.

kubectl得到荚宽-o

NAME       READY  STATUS RESTARTS AGE  IP    NODE 
echoserver-848b75d85-9fx7r 1/1  Running 3   6d  192.168.70.2 k8swrksv01 
nginx-demo-85cc49574c-wv2b9 1/1  Running 3   6d  192.168.2.2 k8swrksv02

kubectl得到SVC -o宽

NAME   TYPE  CLUSTER-IP  EXTERNAL-IP PORT(S)   AGE  SELECTOR 
clusterip  ClusterIP 172.16.39.77  <none>  80/TCP   6d  run=echoserver 
kubernetes  ClusterIP 172.16.0.1  <none>  443/TCP   10d  <none> 
nginx-cluster ClusterIP 172.16.236.159 <none>  8090/TCP   6d  app=nginx-demo 
nginx-service NodePort 172.16.199.69 <none>  8090:31659/TCP 6d  app=nginx-demo 
nodeport  NodePort 172.16.38.40  <none>  80:31317/TCP  6d  run=echoserver

netstat的-ntlp

tcp  0  0 127.0.0.1:10248   0.0.0.0:*    LISTEN  1963/kubelet 
tcp  0  0 127.0.0.1:10249   0.0.0.0:*    LISTEN  2202/kube-proxy 
tcp  0  0 127.0.0.1:4243   0.0.0.0:*    LISTEN  1758/dockerd 
tcp  0  0 0.0.0.0:22    0.0.0.0:*    LISTEN  996/sshd 
tcp6  0  0 :::4194     :::*     LISTEN  1963/kubelet 
tcp6  0  0 :::10250    :::*     LISTEN  1963/kubelet 
tcp6  0  0 :::31659    :::*     LISTEN  2202/kube-proxy 
tcp6  0  0 :::10255    :::*     LISTEN  1963/kubelet 
tcp6  0  0 :::10256    :::*     LISTEN  2202/kube-proxy 
tcp6  0  0 :::31317    :::*     LISTEN  2202/kube-proxy 
tcp6  0  0 :::22     :::*     LISTEN  996/sshd

iptables-save命令

*nat 
:PREROUTING ACCEPT [0:0] 
:INPUT ACCEPT [0:0] 
:OUTPUT ACCEPT [0:0] 
:POSTROUTING ACCEPT [0:0] 
:DOCKER - [0:0] 
:KUBE-MARK-DROP - [0:0] 
:KUBE-MARK-MASQ - [0:0] 
:KUBE-NODEPORTS - [0:0] 
:KUBE-POSTROUTING - [0:0] 
:KUBE-SEP-AZ4EGFEAU4RTSLJO - [0:0] 
:KUBE-SEP-C7HQKKO26GIFOZZM - [0:0] 
:KUBE-SEP-EWKNS2YCPXGJCXDC - [0:0] 
:KUBE-SEP-LQVPUPFGW6BWATIP - [0:0] 
:KUBE-SEP-OMMOFZ27GPKZ4OPA - [0:0] 
:KUBE-SEP-UD3HOGDD5NDLNY74 - [0:0] 
:KUBE-SERVICES - [0:0] 
:KUBE-SVC-CQNAS6RSUGJF2C2D - [0:0] 
:KUBE-SVC-GKN7Y2BSGW4NJTYL - [0:0] 
:KUBE-SVC-NPX46M4PTMTKRN6Y - [0:0] 
:KUBE-SVC-XP7QDA4CRQ2QA33W - [0:0] 
:KUBE-SVC-Z5P6OMNAEVLAQUTS - [0:0] 
-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES 
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER 
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES 
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER 
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING 
-A POSTROUTING -s 192.168.2.0/24 ! -o docker0 -j MASQUERADE 
-A POSTROUTING -s 192.168.0.0/16 -d 192.168.0.0/16 -j RETURN 
-A POSTROUTING -s 192.168.0.0/16 ! -d 224.0.0.0/4 -j MASQUERADE 
-A POSTROUTING ! -s 192.168.0.0/16 -d 192.168.2.0/24 -j RETURN 
-A POSTROUTING ! -s 192.168.0.0/16 -d 192.168.0.0/16 -j MASQUERADE 
-A DOCKER -i docker0 -j RETURN 
-A KUBE-MARK-DROP -j MARK --set-xmark 0x8000/0x8000 
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000 
-A KUBE-NODEPORTS -p tcp -m comment --comment "default/nginx-service:" -m tcp --dport 31659 -j KUBE-MARK-MASQ 
-A KUBE-NODEPORTS -p tcp -m comment --comment "default/nginx-service:" -m tcp --dport 31659 -j KUBE-SVC-GKN7Y2BSGW4NJTYL 
-A KUBE-NODEPORTS -p tcp -m comment --comment "default/nodeport:" -m tcp --dport 31317 -j KUBE-MARK-MASQ 
-A KUBE-NODEPORTS -p tcp -m comment --comment "default/nodeport:" -m tcp --dport 31317 -j KUBE-SVC-XP7QDA4CRQ2QA33W 
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -m mark --mark 0x4000/0x4000 -j MASQUERADE 
-A KUBE-SEP-AZ4EGFEAU4RTSLJO -s 192.168.70.2/32 -m comment --comment "default/clusterip:" -j KUBE-MARK-MASQ 
-A KUBE-SEP-AZ4EGFEAU4RTSLJO -p tcp -m comment --comment "default/clusterip:" -m tcp -j DNAT --to-destination 192.168.70.2:8080 
-A KUBE-SEP-C7HQKKO26GIFOZZM -s 192.168.70.2/32 -m comment --comment "default/nodeport:" -j KUBE-MARK-MASQ 
-A KUBE-SEP-C7HQKKO26GIFOZZM -p tcp -m comment --comment "default/nodeport:" -m tcp -j DNAT --to-destination 192.168.70.2:8080 
-A KUBE-SEP-EWKNS2YCPXGJCXDC -s 10.1.1.25/32 -m comment --comment "default/kubernetes:https" -j KUBE-MARK-MASQ 
-A KUBE-SEP-EWKNS2YCPXGJCXDC -p tcp -m comment --comment "default/kubernetes:https" -m recent --set --name KUBE-SEP-EWKNS2YCPXGJCXDC --mask 255.255.255.255 --rsource -m tcp -j DNAT --to-destination 10.1.1.25:6443 
-A KUBE-SEP-LQVPUPFGW6BWATIP -s 192.168.2.2/32 -m comment --comment "default/nginx-service:" -j KUBE-MARK-MASQ 
-A KUBE-SEP-LQVPUPFGW6BWATIP -p tcp -m comment --comment "default/nginx-service:" -m tcp -j DNAT --to-destination 192.168.2.2:80 
-A KUBE-SEP-OMMOFZ27GPKZ4OPA -s 10.1.1.24/32 -m comment --comment "default/kubernetes:https" -j KUBE-MARK-MASQ 
-A KUBE-SEP-OMMOFZ27GPKZ4OPA -p tcp -m comment --comment "default/kubernetes:https" -m recent --set --name KUBE-SEP-OMMOFZ27GPKZ4OPA --mask 255.255.255.255 --rsource -m tcp -j DNAT --to-destination 10.1.1.24:6443 
-A KUBE-SEP-UD3HOGDD5NDLNY74 -s 192.168.2.2/32 -m comment --comment "default/nginx-cluster:" -j KUBE-MARK-MASQ 
-A KUBE-SEP-UD3HOGDD5NDLNY74 -p tcp -m comment --comment "default/nginx-cluster:" -m tcp -j DNAT --to-destination 192.168.2.2:80 
-A KUBE-SERVICES -d 172.16.236.159/32 -p tcp -m comment --comment "default/nginx-cluster: cluster IP" -m tcp --dport 8090 -j KUBE-SVC-Z5P6OMNAEVLAQUTS 
-A KUBE-SERVICES -d 172.16.199.69/32 -p tcp -m comment --comment "default/nginx-service: cluster IP" -m tcp --dport 8090 -j KUBE-SVC-GKN7Y2BSGW4NJTYL 
-A KUBE-SERVICES -d 172.16.38.40/32 -p tcp -m comment --comment "default/nodeport: cluster IP" -m tcp --dport 80 -j KUBE-SVC-XP7QDA4CRQ2QA33W 
-A KUBE-SERVICES -d 172.16.39.77/32 -p tcp -m comment --comment "default/clusterip: cluster IP" -m tcp --dport 80 -j KUBE-SVC-CQNAS6RSUGJF2C2D 
-A KUBE-SERVICES -d 172.16.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-SVC-NPX46M4PTMTKRN6Y 
-A KUBE-SERVICES -m comment --comment "kubernetes service nodeports; NOTE: this must be the last rule in this chain" -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS 
-A KUBE-SVC-CQNAS6RSUGJF2C2D -m comment --comment "default/clusterip:" -j KUBE-SEP-AZ4EGFEAU4RTSLJO 
-A KUBE-SVC-GKN7Y2BSGW4NJTYL -m comment --comment "default/nginx-service:" -j KUBE-SEP-LQVPUPFGW6BWATIP 
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m recent --rcheck --seconds 10800 --reap --name KUBE-SEP-OMMOFZ27GPKZ4OPA --mask 255.255.255.255 --rsource -j KUBE-SEP-OMMOFZ27GPKZ4OPA 
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m recent --rcheck --seconds 10800 --reap --name KUBE-SEP-EWKNS2YCPXGJCXDC --mask 255.255.255.255 --rsource -j KUBE-SEP-EWKNS2YCPXGJCXDC 
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-OMMOFZ27GPKZ4OPA 
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -j KUBE-SEP-EWKNS2YCPXGJCXDC 
-A KUBE-SVC-XP7QDA4CRQ2QA33W -m comment --comment "default/nodeport:" -j KUBE-SEP-C7HQKKO26GIFOZZM 
-A KUBE-SVC-Z5P6OMNAEVLAQUTS -m comment --comment "default/nginx-cluster:" -j KUBE-SEP-UD3HOGDD5NDLNY74 
COMMIT 
*filter 
:INPUT ACCEPT [40:14606] 
:FORWARD DROP [0:0] 
:OUTPUT ACCEPT [42:6275] 
:DOCKER - [0:0] 
:DOCKER-ISOLATION - [0:0] 
:DOCKER-USER - [0:0] 
:KUBE-FIREWALL - [0:0] 
:KUBE-SERVICES - [0:0] 
-A INPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES 
-A INPUT -j KUBE-FIREWALL 
-A FORWARD -j DOCKER-USER 
-A FORWARD -j DOCKER-ISOLATION 
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT 
-A FORWARD -o docker0 -j DOCKER 
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT 
-A FORWARD -i docker0 -o docker0 -j ACCEPT 
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES 
-A OUTPUT -j KUBE-FIREWALL 
-A DOCKER-ISOLATION -j RETURN 
-A DOCKER-USER -j RETURN 
-A KUBE-FIREWALL -m comment --comment "kubernetes firewall for dropping marked packets" -m mark --mark 0x8000/0x8000 -j DROP 
COMMIT

来源

2017-10-10 Y.Yama

+0

您与哪个提供商合作？ GKE？ – iamnat

+0

哦，我忘了写。不使用云。并没有使用kubeadm，防火墙（ufw）被禁用。 –

回答

1

这是由F的默认DROP造成的ORWARD链（反过来是由码头造成的）。

如果您将iptables -A FORWARD -j ACCEPT规则添加到您的节点，您可以再次看到它工作。

k8s问题在这里：https://github.com/kubernetes/kubernetes/issues/39823但实际的修复在这里https://github.com/kubernetes/kubernetes/pull/52569（预计在1.9）。

来源

2017-12-03 12:56:50 Farcaller

+0

谢谢！ ！ 这是原因。尽管我摧毁了发布时间的环境，但我试图在另一个环境中创建它，并且毫无问题地解决它！ –

-1

你可以先检查节点端口的端口是否打开，

使用

netstat -ntlp

检查，如果是，那也许有些问题用的iptable或路由，

如果没有，请检查防火墙或其他问题

好lu ck

来源

2017-10-14 08:48:02 sam

+0

谢谢！ 添加了netstat和iptables-save的结果。 此端口已打开。 并检查了iptables的内容。但是，我认为这个内容是正确的... –

+0

你可以遵循这个规则，首先是'-A KUBE-NODEPORTS -p tcp -m comment --comment“default/nginx-service：”-m tcp --dport 31659 -j KUBE-SVC-GKN7Y2BSGW4NJTYL“表示31659将会转到'KUBE-SVC-GKN7Y2BSGW4NJTYL'，然后是'-A KUBE-SVC-GKN7Y2BSGW4NJTYL -m注释 - 注释“default/nginx-service：”-j KUBE-SEP-LQVPUPFGW6BWATIP'，它告诉你它会转移到这个地方，最后'-A KUBE-SEP-LQVPUPFGW6BWATIP -p tcp -m注释 - 注释“default/nginx-service：”-m tcp -j DNAT - to-destination 192.168.2.2：80'，告诉您的请求将转到192.168.2.2:80，并检查该吊舱是否相同？ – sam

+0

并且你可以运行一个ubuntu来检查集群ip端口是否工作呢？如果你按照这一步找到解决方案，请给我一个肯定的评论，我不知道为什么有人给我负面的评论，说问题的解决方案是总是喜欢这个.....我不是上帝可以直接找到答案.... – sam

0

附加信息。 东西应该阻止端口，但未知...

吊舱正在运行的节点

的nmap 10.1.1.27 -p31000-32000

Not shown: 999 closed ports 
PORT  STATE SERVICE 
31317/tcp open unknown 
31659/tcp open unknown

其他节点

NMAP 10.1 .1.27 -p31000-32000

Not shown: 999 closed ports 
PORT  STATE SERVICE 
31317/tcp filtered unknown 
31659/tcp filtered unknown

来源

2017-10-19 00:06:02

0

Farcaller是正确的。我们在运行firewalld的centos上遇到同样的情况。

在我们升级到k8s 1.9之前，我们添加了以下firewalld规则。该规则类似于k8中的kube-proxy创建的规则1.9

#!/bin/bash 
# follows https://github.com/kubernetes/kubernetes/pull/52569 introduced in k8s 1.9 
# required to support nodeport services routing from all nodes in the cluster when the firewall is turned on. 
# KUBE-MARK-MASQ corresponds to kube-proxy --iptables-masquerade-bit=14, which is the default. 
KUBE_MARK_MASQ="0x4000/0x4000" 
firewall-cmd --permanent --direct --add-rule ipv4 filter FORWARD 3 -m mark --mark "$KUBE_MARK_MASQ" -j ACCEPT

来源

2018-02-16 08:59:27

相关问题

 相关问题