普罗米修斯 - Kubernetes RBAC

Question

我我GKE API服务器升级到1.6，我在升级节点到1.6的过程中，却遇到了一个问题...

我有一个普罗米修斯服务器（1.5版本。 2）运行在由Kubernetes部署管理的Pod上，其中有一些节点运行版本1.5.4 Kubelet，其中一个新节点运行1.6。

Prometheus无法连接到新节点 - 指标端点返回401未经授权。

这似乎是一个RBAC问题，但我不知道如何继续。我找不到有关Prometheus服务器需要什么角色的文档，甚至不知道如何将它们授予服务器。

从coreos /普罗米修斯运营商回购我能拼凑出我所料工作的配置：

apiVersion: v1 
kind: ServiceAccount 
metadata: 
    name: prometheus 
--- 

apiVersion: rbac.authorization.k8s.io/v1beta1 
kind: ClusterRole 
metadata: 
    name: prometheus 
rules: 
- apiGroups: [""] 
    resources: 
    - nodes 
    - services 
    - endpoints 
    - pods 
    verbs: ["get", "list", "watch"] 
- apiGroups: [""] 
    resources: 
    - configmaps 
    verbs: ["get"] 
- nonResourceURLs: ["/metrics"] 
    verbs: ["get"] 
--- 

apiVersion: rbac.authorization.k8s.io/v1beta1 
kind: ClusterRoleBinding 
metadata: 
    name: prometheus 
roleRef: 
    apiGroup: rbac.authorization.k8s.io 
    kind: ClusterRole 
    name: prometheus 
subjects: 
- kind: ServiceAccount 
    name: prometheus 
    namespace: default 
--- 

apiVersion: v1 
kind: ServiceAccount 
metadata: 
    name: prometheus 
    namespace: default 
secrets: 
- name: prometheus-token-xxxxx 

--- 
apiVersion: extensions/v1beta1 
kind: Deployment 
metadata: 
    labels: 
    app: prometheus-prometheus 
    component: server 
    release: prometheus 
    name: prometheus-server 
    namespace: default 
spec: 
    replicas: 1 
    selector: 
    matchLabels: 
     app: prometheus-prometheus 
     component: server 
     release: prometheus 
    strategy: 
    rollingUpdate: 
     maxSurge: 1 
     maxUnavailable: 1 
    type: RollingUpdate 
    template: 
    metadata: 
     labels: 
     app: prometheus-prometheus 
     component: server 
     release: prometheus 
    spec: 
     dnsPolicy: ClusterFirst 
     restartPolicy: Always 
     schedulerName: default-scheduler 
     serviceAccount: prometheus 
     serviceAccountName: prometheus 
     ... 

但是普罗米修斯还是日渐401S。

更新：看起来像乔丹所说的kubernetes认证问题。在此查看新的，更具针对性的问题; https://serverfault.com/questions/843751/kubernetes-node-metrics-endpoint-returns-401

Answer 1

根据对@ JorritSalverda的票证的讨论; https://github.com/prometheus/prometheus/issues/2606#issuecomment-294869099

由于GKE不允许您获得允许您使用kubelet进行身份验证的客户端证书，因此GKE用户的最佳解决方案似乎使用kubernetes API服务器作为代理请求节点。

要做到这一点（引用@JorritSalverda）;

“对于内部GKE运行我的普罗米修斯的服务器我现在已经将其与以下重新贴标签运行：

relabel_configs: 
- action: labelmap 
    regex: __meta_kubernetes_node_label_(.+) 
- target_label: __address__ 
    replacement: kubernetes.default.svc.cluster.local:443 
- target_label: __scheme__ 
    replacement: https 
- source_labels: [__meta_kubernetes_node_name] 
    regex: (.+) 
    target_label: __metrics_path__ 
    replacement: /api/v1/nodes/${1}/proxy/metrics 

并绑定到普罗米修斯使用的服务帐户以下ClusterRole：

apiVersion: rbac.authorization.k8s.io/v1beta1 
kind: ClusterRole 
metadata: 
    name: prometheus 
rules: 
- apiGroups: [""] 
    resources: 
    - nodes 
    - nodes/proxy 
    - services 
    - endpoints 
    - pods 
    verbs: ["get", "list", "watch"] 

因为在RBAC失败的情况下，GKE集群仍然具有ABAC后备箱我并不是100％确定，但它涵盖了所有必需的权限。

Answer 2

401意味着未经认证，这意味着它不是一个RBAC问题。我相信GKE不再允许匿名访问1.6版的kubelet。你用什么凭证来认证kubelet？

Answer 3

这是我h ave正在为角色定义和绑定工作。

apiVersion: rbac.authorization.k8s.io/v1beta1 
 
kind: ClusterRole 
 
metadata: 
 
    name: prometheus 
 
rules: 
 
- apiGroups: [""] 
 
    resources: 
 
    - nodes 
 
    - services 
 
    - endpoints 
 
    - pods 
 
    verbs: ["get", "list", "watch"] 
 
- nonResourceURLs: ["/metrics"] 
 
    verbs: ["get"] 
 
--- 
 
apiVersion: v1 
 
kind: ServiceAccount 
 
metadata: 
 
    name: prometheus 
 
    namespace: default 
 
--- 
 
apiVersion: rbac.authorization.k8s.io/v1beta1 
 
kind: ClusterRoleBinding 
 
metadata: 
 
    name: prometheus 
 
roleRef: 
 
    apiGroup: rbac.authorization.k8s.io 
 
    kind: ClusterRole 
 
    name: prometheus 
 
subjects: 
 
- kind: ServiceAccount 
 
    name: prometheus 
 
    namespace: default