Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

http/1.1 protocol error: HPE_INVALID_METHOD #54016

Open
2 tasks done
tirelibirefe opened this issue Nov 21, 2024 · 6 comments
Open
2 tasks done

http/1.1 protocol error: HPE_INVALID_METHOD #54016

tirelibirefe opened this issue Nov 21, 2024 · 6 comments
Labels
area/environments area/networking area/security

Comments

@tirelibirefe
Copy link

tirelibirefe commented Nov 21, 2024 •
edited
Loading

Is this the right place to submit this?

  • This is not a security vulnerability or a crashing bug
  • This is not a question about how to use Istio

Bug Description

Hello,
I am trying to implement secure istio gateway. The traffic will access to istio-ingress gateway through ALB. My configs are below:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  namespace: istio-ingress
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-service
  namespace: istio-ingress
spec:
  selector:
    app: nginx
  ports:
    - name: http
      port: 80
      targetPort: 80
  type: ClusterIP
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: common-gateway
  namespace: istio-ingress
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 443
      name: https
      protocol: HTTPS
    hosts:
      - "*"
    tls:
      mode: SIMPLE
      credentialName: istio-ingress/mycompany-certs
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: nginx-virtualservice
  namespace: istio-ingress
spec:
  hosts:
    - nginx
  gateways:
    - common-gateway
  http:
  - match:
    - uri:
        prefix: /
    route:
    - destination:
        host: nginx-service
        port:
          number: 80

my AWS ALB Ingress Config:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: istio-gateway-external
  namespace: istio-ingress
  annotations:
    alb.ingress.kubernetes.io/healthcheck-path: /healthz/ready
    alb.ingress.kubernetes.io/healthcheck-port: status-port
    alb.ingress.kubernetes.io/healthcheck-protocol: HTTP
    alb.ingress.kubernetes.io/backend-protocol: HTTPS
    alb.ingress.kubernetes.io/backend-protocol-version: HTTP2
    alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:eu-central-1:123123123123:certificate/1d2d0756-39c2-4378-8e42-ef997975b444
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: instance
    alb.ingress.kubernetes.io/load-balancer-name: istio-gateway-external
    alb.ingress.kubernetes.io/actions.ssl-redirect: |
      {
        "Type": "redirect",
        "RedirectConfig": {
        "Protocol": "HTTPS",
        "Port": "443",
        "StatusCode": "HTTP_301"
        }
      }
spec:
  ingressClassName: alb
  rules:
  - http:
      paths:
      - backend:
          service:
            name: ssl-redirect
            port:
              name: use-annotation
        path: /
        pathType: Prefix
      - backend:
          service:
            name: istio-ingressgateway
            port:
              number: 15021
        path: /
        pathType: Prefix
      - backend:
          service:
            name: istio-ingressgateway
            port:
              number: 443
        path: /
        pathType: Prefix

I create certs:

export DOMAIN_NAME=mycompany

openssl req -x509 -sha256 -nodes -days 365 \
  -newkey rsa:2048 -subj '/O='$DOMAIN_NAME' Inc./CN='$DOMAIN_NAME'' \
  -keyout $DOMAIN_NAME.key -out $DOMAIN_NAME.crt

openssl req -out \*.$DOMAIN_NAME.csr -newkey rsa:2048 \
  -nodes -keyout \*.$DOMAIN_NAME.key \
  -subj "/CN=*.$DOMAIN_NAME/O=apps from $DOMAIN_NAME"

openssl x509 -req -days 365 -CA $DOMAIN_NAME.crt \
  -CAkey $DOMAIN_NAME.key -set_serial 0 \
  -in \*.$DOMAIN_NAME.csr -out \*.$DOMAIN_NAME.crt

kubectl create secret tls mycompany-certs -n istio-ingress \
  --key $DOMAIN_NAME.key \
  --cert $DOMAIN_NAME.crt

Some checks:

tmp-shell:~# curl https://asdfadsfads.mydomain.com
<html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
</body>
</html>
tmp-shell:~# curl https://nginx.mydomain.com
<html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
</body>
</html>
tmp-shell:~# curl -I -k -sS --cert /opt/secret/tls.crt --key /opt/secret/tls.key -H \
  'Host: nginx.mydomain.com' https://istio-ingressgateway.istio-ingress.svc 

HTTP/2 404 
date: Thu, 21 Nov 2024 18:39:27 GMT
server: istio-envoy

When I attempt to access app from outside of the cluster I get 502 bad gateway error.
When I attempt to access app from inside of the cluster I get 404 bad gateway error.

Could you please advise what I am missing?

Thanks&Regards

Version

client version: 1.24.0
base / istiod / gateway : 1.23.3
I use Helm Chart 1.23.3 to deploy istio

Additional Information

istiod namespace> istio-system
istio-gateway > istio-ingress

istioctl analyze -n demoapps doesn't find any problems.

My loadbalancer healthchecks are ok.

My istio ingress gateway uses "NodePort" service.

When I try the same architecture by using nginx ingress controller (without tls), it works.

My EKS has following rules:

...
  node_security_group_additional_rules = {
    ingress_cluster_15017 = {
      description                   = "Cluster API to node groups"
      protocol                      = "tcp"
      from_port                     = 15017
      to_port                       = 15017
      type                          = "ingress"
      source_cluster_security_group = true
    }
    ingress_15012 = {
      description                   = "Cluster API to nodes ports/protocols"
      protocol                      = "TCP"
      from_port                     = 15012
      to_port                       = 15012
      type                          = "ingress"
      source_cluster_security_group = true
    }
  }
...

My istio-ingressgateway produces the following errors:

2024-11-21T18:37:42.222413Z	debug	envoy http external/envoy/source/common/http/filter_manager.cc:1075	[Tags: "ConnectionId":"2110","StreamId":"13274701494911436017"] Sending local reply with details http1.codec_error	thread=21
2024-11-21T18:37:42.222830Z	debug	envoy http external/envoy/source/common/http/conn_manager_impl.cc:1894	[Tags: "ConnectionId":"2110","StreamId":"13274701494911436017"] stream reset: reset reason: local reset, response details: http1.codec_error	thread=21
2024-11-21T18:37:42.222899Z	debug	envoy http external/envoy/source/common/http/conn_manager_impl.cc:431	[Tags: "ConnectionId":"2110"] dispatch error: http/1.1 protocol error: HPE_INVALID_METHOD	thread=21
@istio istio locked and limited conversation to collaborators Nov 21, 2024
@istio istio unlocked this conversation Nov 21, 2024
@howardjohn
Copy link
Member

howardjohn commented Nov 21, 2024 •
edited
Loading

When I attempt to access app from inside of the cluster I get 404 bad gateway error.

This is because you are only overriding the Host header, not the SNI (TLS) name.

@tirelibirefe
Copy link
Author

When I attempt to access app from inside of the cluster I get 404 bad gateway error.

excuse me pls; wrong sentence; correcting:
When I attempt to access app from inside of the cluster I get 404 not found error.

@tirelibirefe
Copy link
Author

This is because you are only overriding the Host header, not the SNI (TLS) name.

Sorry, I didn't understand, what do you mean? I've used;

    hosts:
      - "*"

@tirelibirefe
Copy link
Author

tirelibirefe commented Nov 21, 2024 •
edited
Loading

This is because you are only overriding the Host header, not the SNI (TLS) name.

Actually, I have no purpose like "to override" or "not to override". Should I preserve the header? How can do it or how can I set it? Could you please advise?

routing.http.preserve_host_header.enabled ?

@tirelibirefe
Copy link
Author

I tried routing.http.preserve_host_header.enabled: true nothing changed.

@tirelibirefe
Copy link
Author

Further logs:

istioctl proxy-config routes -n demoapps nginx-deployment-54b9c68f67-nr5rv

NAME                                                                    VHOST NAME                                                              DOMAINS                                               MATCH                  VIRTUAL SERVICE
aws-load-balancer-webhook-service.kube-system.svc.cluster.local:443     aws-load-balancer-webhook-service.kube-system.svc.cluster.local:443     *                                                     /*                     
80                                                                      istio-ingressgateway.istio-ingress.svc.cluster.local:80                 istio-ingressgateway.istio-ingress, 172.20.101.67     /*                     
80                                                                      nginx-service.demoapps.svc.cluster.local:80                             nginx-service, nginx-service.demoapps + 1 more...     /*                     
15010                                                                   istiod.istio-system.svc.cluster.local:15010                             istiod.istio-system, 172.20.110.27                    /*                     
cert-manager-webhook.cert-manager.svc.cluster.local:9402                cert-manager-webhook.cert-manager.svc.cluster.local:9402                *                                                     /*                     
eks-extension-metrics-api.kube-system.svc.cluster.local:443             eks-extension-metrics-api.kube-system.svc.cluster.local:443             *                                                     /*                     
istio-ingressgateway.istio-ingress.svc.cluster.local:15021              istio-ingressgateway.istio-ingress.svc.cluster.local:15021              *                                                     /*                     
9402                                                                    cert-manager-cainjector.cert-manager.svc.cluster.local:9402             cert-manager-cainjector.cert-manager, 172.20.4.68     /*                     
9402                                                                    cert-manager-webhook.cert-manager.svc.cluster.local:9402                cert-manager-webhook.cert-manager, 172.20.84.204      /*                     
15014                                                                   istiod.istio-system.svc.cluster.local:15014                             istiod.istio-system, 172.20.110.27                    /*                     
kube-dns.kube-system.svc.cluster.local:9153                             kube-dns.kube-system.svc.cluster.local:9153                             *                                                     /*                     
InboundPassthroughCluster                                               inbound|http|0                                                          *                                                     /*                     
inbound|80||                                                            inbound|http|80                                                         *                                                     /*                     
                                                                        backend                                                                 *                                                     /stats/prometheus*     
InboundPassthroughCluster                                               inbound|http|0                                                          *                                                     /*                     
                                                                        backend                                                                 *                                                     /healthz/ready*        
inbound|80||                                                            inbound|http|80                                                         *                                                     /*  

istioctl proxy-config listeners nginx-deployment-54b9c68f67-nr5rv -n demoapps

ADDRESSES      PORT  MATCH                                                                                         DESTINATION
172.20.0.10    53    ALL                                                                                           Cluster: outbound|53||kube-dns.kube-system.svc.cluster.local
0.0.0.0        80    Trans: raw_buffer; App: http/1.1,h2c                                                          Route: 80
0.0.0.0        80    ALL                                                                                           PassthroughCluster
172.20.0.1     443   ALL                                                                                           Cluster: outbound|443||kubernetes.default.svc.cluster.local
172.20.101.67  443   ALL                                                                                           Cluster: outbound|443||istio-ingressgateway.istio-ingress.svc.cluster.local
172.20.110.27  443   ALL                                                                                           Cluster: outbound|443||istiod.istio-system.svc.cluster.local
172.20.156.69  443   Trans: raw_buffer; App: http/1.1,h2c                                                          Route: eks-extension-metrics-api.kube-system.svc.cluster.local:443
172.20.156.69  443   ALL                                                                                           Cluster: outbound|443||eks-extension-metrics-api.kube-system.svc.cluster.local
172.20.187.174 443   ALL                                                                                           Cluster: outbound|443||metrics-server.kube-system.svc.cluster.local
172.20.251.196 443   Trans: raw_buffer; App: http/1.1,h2c                                                          Route: aws-load-balancer-webhook-service.kube-system.svc.cluster.local:443
172.20.251.196 443   ALL                                                                                           Cluster: outbound|443||aws-load-balancer-webhook-service.kube-system.svc.cluster.local
172.20.84.204  443   ALL                                                                                           Cluster: outbound|443||cert-manager-webhook.cert-manager.svc.cluster.local
172.20.0.10    9153  Trans: raw_buffer; App: http/1.1,h2c                                                          Route: kube-dns.kube-system.svc.cluster.local:9153
172.20.0.10    9153  ALL                                                                                           Cluster: outbound|9153||kube-dns.kube-system.svc.cluster.local
0.0.0.0        9402  Trans: raw_buffer; App: http/1.1,h2c                                                          Route: 9402
0.0.0.0        9402  ALL                                                                                           PassthroughCluster
172.20.152.34  9402  ALL                                                                                           Cluster: outbound|9402||cert-manager.cert-manager.svc.cluster.local
172.20.84.204  9402  Trans: raw_buffer; App: http/1.1,h2c                                                          Route: cert-manager-webhook.cert-manager.svc.cluster.local:9402
172.20.84.204  9402  ALL                                                                                           Cluster: outbound|9402||cert-manager-webhook.cert-manager.svc.cluster.local
0.0.0.0        15001 ALL                                                                                           PassthroughCluster
0.0.0.0        15001 Addr: *:15001                                                                                 Non-HTTP/Non-TCP
0.0.0.0        15006 Addr: *:15006                                                                                 Non-HTTP/Non-TCP
0.0.0.0        15006 Trans: tls; App: istio-http/1.0,istio-http/1.1,istio-h2                                       InboundPassthroughCluster
0.0.0.0        15006 Trans: raw_buffer; App: http/1.1,h2c                                                          InboundPassthroughCluster
0.0.0.0        15006 Trans: tls; App: TCP TLS                                                                      InboundPassthroughCluster
0.0.0.0        15006 Trans: raw_buffer                                                                             InboundPassthroughCluster
0.0.0.0        15006 Trans: tls                                                                                    InboundPassthroughCluster
0.0.0.0        15006 Trans: tls; App: istio,istio-peer-exchange,istio-http/1.0,istio-http/1.1,istio-h2; Addr: *:80 Cluster: inbound|80||
0.0.0.0        15006 Trans: raw_buffer; Addr: *:80                                                                 Cluster: inbound|80||
0.0.0.0        15010 Trans: raw_buffer; App: http/1.1,h2c                                                          Route: 15010
0.0.0.0        15010 ALL                                                                                           PassthroughCluster
172.20.110.27  15012 ALL                                                                                           Cluster: outbound|15012||istiod.istio-system.svc.cluster.local
0.0.0.0        15014 Trans: raw_buffer; App: http/1.1,h2c                                                          Route: 15014
0.0.0.0        15014 ALL                                                                                           PassthroughCluster
0.0.0.0        15021 ALL                                                                                           Inline Route: /healthz/ready*
172.20.101.67  15021 Trans: raw_buffer; App: http/1.1,h2c                                                          Route: istio-ingressgateway.istio-ingress.svc.cluster.local:15021
172.20.101.67  15021 ALL                                                                                           Cluster: outbound|15021||istio-ingressgateway.istio-ingress.svc.cluster.local
0.0.0.0        15090 ALL                                                                                           Inline Route: /stats/prometheus*

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/environments area/networking area/security
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@howardjohn @tirelibirefe @istio-policy-bot