I apologize in advance for this long plea. I know that people have asked similar versions of this question before. I've read them. I've been searching the docs and SO and GitHub and even trying to read through test files for days but cannot seem to get it to work. There are so many different variables and API versions and I'm struggling to pinpoint the problem.

I'm looking for the absolute minimal example to invoke this example gRPC server through an Istio Gateway using a VirtualService on Minikube. Here's what I've got:

Setup:

# Use enough resources for Istio.
minikube start --memory=16384 --cpus=4
# Set up Istio in Ambient mode.
istioctl install --set profile=ambient --skip-confirmation
# Set up Gateway API.
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.0/standard-install.yaml

And here are all the resources:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: foo-gateway
spec:
  gatewayClassName: istio
  listeners:
    - name: foo-grpc
      hostname: "localhost"
      port: 50051
      protocol: HTTP
---
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: foo-route
spec:
  gateways:
    - foo-gateway
  hosts:
    - "localhost"
  http:
    - name: foo-route
      route:
        - destination:
            host: foo-service.default.svc.cluster.local
---
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: reviews-destination
spec:
  host: foo-service.default.svc.cluster.local
---
apiVersion: v1
kind: Service
metadata:
  name: foo-service
  labels:
    app: foo-app
spec:
  ports:
    - name: grpc
      port: 50051
      appProtocol: k8s.io/grpc
  selector:
    app: foo-app
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: foo-deployment
  labels:
    app: foo-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: foo-app
  template:
    metadata:
      labels:
        app: foo-app
    spec:
      containers:
        - name: foo-container
          image: 'docker.io/grpc/java-example-hostname'
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 50051

I can invoke the service directly:

kubectl port-forward svc/foo-service 9090:50051
grpcurl -plaintext -format text -d 'name: "Jimbo"' localhost:9090 helloworld.Greeter/SayHello

But when I instead port-forward to svc/foo-gateway-istio, I get this error:

ERROR:
  Code: Unimplemented
  Message: 

I tried checking things out with Kubeshark and found the matching gRPC request, from foo-gateway-istio-77bc48898f-4b98h/envoy to itself (src and dst are the same). All the headers look right. Interestingly, the request's body (it says "text/plain" for some reason) is {"field_1":"Jimbo"} which almost seems to suggest there's some automatic gRPC-JSON transcoding going on, which would be super confusing.

I would very much appreciate tips on debugging problems like this in general. It seems like Istio issues can be difficult to debug in general. Kubeshark is not providing much help here and the logs for the gateway pod (foo-gateway-istio-77bc48898f-4b98h) are stuck at Envoy proxy is ready. It will save me quite a bit of hair if anybody could also point out the small edit(s) to these resource definitions that I'm sure I'm naively missing.