Kubernetes中的Nginx Controller：与上游握手-SSL握手中的对等封闭连接

如何解决Kubernetes中的Nginx Controller：与上游握手-SSL握手中的对等封闭连接

在使用kubeadm构建的本地环境中。群集由主节点和2个工作节点组成。

失败之处：

通过服务类型LoadBalancer将Nginx-ingress Controller暴露于外部，尝试对Kubernetes集群进行TLS终止。

这是公开的服务：

NAME            TYPE           CLUSTER-IP      EXTERNAL-IP    PORT(S)                      
nginx-ingress   LoadBalancer   10.101.5.75     192.168.1.82   80:30745/TCP,443:30092/TCP   
web-service     ClusterIP      10.101.26.176   <none>         8080/TCP

工作原理：

能够从外部使用HTTP端口80访问Web应用程序。

这是集群中的豆荚：

NAME                              READY   STATUS    RESTARTS   AGE   IP          NODE    
nginx-ingress-7c5588544d-4mw9d    1/1     Running   0          38m   10.44.0.2   node1   
web-deployment-7d84778bc6-52pq7   1/1     Running   1          19h   10.44.0.1   node1   
web-deployment-7d84778bc6-9wwmn   1/1     Running   1          19h   10.36.0.2   node2   

TLS终止的测试结果

客户端：

curl -k https://example.com -v
* Rebuilt URL to https://example.com/
*   Trying 192.168.1.82...
* Connected to example.com (192.168.1.82) port 443 (#0)
* found 127 certificates in /etc/ssl/certs/ca-certificates.crt
* found 513 certificates in /etc/ssl/certs
* ALPN,offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
*    server certificate verification SKIPPED
*    server certificate status verification SKIPPED
*    common name: example.com (matched)
*    server certificate expiration date OK
*    server certificate activation date OK
*    certificate public key: RSA
*    certificate version: #1
*    subject: CN=example.com
*    start date: Thu,22 Oct 2020 16:33:49 GMT
*    expire date: Fri,22 Oct 2021 16:33:49 GMT
*    issuer: CN=My Cert Authority
*    compression: NULL
* ALPN,server accepted to use http/1.1
> GET / HTTP/1.1
> Host: example.com
> User-Agent: curl/7.47.0
> Accept: */*
> 
< HTTP/1.1 502 Bad Gateway
< Server: nginx/1.19.3

服务器端（来自nginx-ingress pod的日志）：

GET / HTTP/1.1
Connection: close
Host: example.com
X-Real-IP: 10.32.0.1
X-Forwarded-For: 10.32.0.1
X-Forwarded-Host: example.com
X-Forwarded-Port: 443
X-Forwarded-Proto: https
User-Agent: curl/7.47.0
Accept: */*

2020/10/22 18:38:59 [error] 23#23: *12 peer closed connection in SSL handshake while SSL handshaking  to upstream,client: 10.32.0.1,server: example.com,request: "GET / HTTP/1.1",upstream: "https://10.44.0.1:8081/",host: "example.com"

2020/10/22 18:38:59 [warn] 23#23: *12 upstream server temporarily disabled while SSL handshaking to upstream,host: "example.com"

HTTP/1.1 502 Bad Gateway
Server: nginx/1.19.3

我检查的内容

通过以下链接生成了CA和服务器证书： https://kubernetes.github.io/ingress-nginx/examples/PREREQUISITES/#tls-certificates

检查了nginx-ingress Pod的/ etc / nginx / secrets / default下的服务器证书和服务器密钥，它们看起来正确。这是资源VirtualServer的输出：

NAME         STATE   HOST          IP             PORTS      AGE
vs-example   Valid   example.com   192.168.1.82   [80,443]   121m

虚拟服务器：

apiVersion: k8s.nginx.org/v1
kind: VirtualServer
metadata:
  name: vs-example
  namespace: nginx-ingress
spec:
  host: example.com
  tls:
    secret: example-tls
  upstreams:
  - name: example
    service: web-service
    port: 8080
    tls:
      enable: true
  routes:
  - path: /v2
    action:
      pass: example
  routes:
  - path: /
    action:
      pass: example

秘密

apiVersion: v1
kind: Secret
metadata:
  name: example-tls
  namespace: nginx-ingress
data:
  tls.crt: (..omitted..)
  tls.key: (..omitted..)
type: kubernetes.io/tls

Nginx.conf

以下是从运行的Pod中提取的nginx.conf的摘录：

server {
    # required to support the Websocket protocol in VirtualServer/VirtualServerRoutes
    set $default_connection_header "";
    listen 80 default_server;
    listen 443 ssl default_server;
    ssl_certificate /etc/nginx/secrets/default;
    ssl_certificate_key /etc/nginx/secrets/default;
    server_name _;
    server_tokens "on";

尝试使用HTTPS访问Web应用程序时找不到发生的事情。

解决方法

通过这些更改进行编辑的VirtualServer中的几个错误解决了该问题：

apiVersion: k8s.nginx.org/v1
kind: VirtualServer
metadata:
  name: vs-example
  namespace: nginx-ingress
spec:
  host: example.com
  tls:
    secret: example-tls

  upstreams:
  - name: example
    service: web-service
    port: 8080
#    tls:            --> this caused the SSL issue in the upstream
#      enable: true

  routes:
  - path: /
    action:
      pass: example
  - path: /v1
    action:
      pass: example

现在可以使用HTTPS访问Web应用程序