Unable to gRPCurl a gRPC deployment

Troubleshooting and help
1

Heard about Koyeb on the Cup O’ Go podcast.

I’m building a gRPC health check service (https://ack.al) and have a known-working gRPC healthcheck server that implements the gRPC Health Check protocol.

I deployed the container to Koyeb and it deployed successfully, first time. I found the UI unfamiliar and initially didn’t make the container’s (default|50051) port public.

I’m unable to gRPCurl the service’s endpoint.

The UI Dashboard shows that the app’s healthy and the app logs are expected.

I was unclear whether the service would be published under 443 or 50051 but think 443 is correct. I’m able to query the endpoint’s TLS:

openssl s_client -connect ${APP}.koyeb.app:443 2>/dev/null \
| openssl x509 -noout -dates
notBefore=Jun  1 12:10:38 2023 GMT
notAfter=Aug 30 12:10:37 2023 GMT

I’ve tried:

grpcurl \
-vv \
-connect-timeout 10 \
${APP}.koyeb.app:443 \
grpc.health.v1.Health/Check

The connection doesn’t return|timeout.

Is there any way to inspect request logs?

The service assumes the proxy will terminate TLS and expects a plaintext gRPC connection.

I was unable to find a gRPC tutorial in your docs but found e.g. End-to-end gRPC ….

Any guidance on what I’m doing wrong would be appreciated.

2

Hello,

I think this is because the reflect api seems to not be available through our load balancer layer (cloudflare), the connection seems to keep hanging. We will try to investigate with them why.

You can directly use it using a go client with a proto and also using grpcurl but you need to provide a proto file.

Example with this project GitHub - tsaikd/go-grpc-echo: golang grpc echo example

grpcurl -proto pb/echo.proto <service>.koyeb.app:443  go_grpc_echo_pb.Echo.Send                                                                                                                                                                                        

{
  "timestamp": "1687423038",
  "hostname": "737d66cb",
...
}

I hope this help!

Bastien

1 Like
3

I am able to repro this result, thank you!

I thought that reflection would only be involved for `list`'ing services|methods and not for direct method invocation e.g. `grpc.health.v1.Health/Check`: 
# Requires reflection
grpcurl {service}.koyeb.app:443 list

# Requires reflection!?
grpcurl {service}.koyeb.app:443 grpc.health.v1.Health/Check

I’m so used to using gRPC reflection that I’d become unfamiliar with how gRPC behaves without it. I just confirmed that reflection is required even to invoke a specific method.

4

Hello @bastien, do you have some update on this reflection issue? Thks :slight_smile:

5

So far Cloudflare has some hypotheses on why it would be blocked but we didn’t confirm nor reproduce in an isolated setup.

Basically, we are not able to enable at the same time the WAF and the gRPC reflect. The buffering of the request for inspection by the WAF might be the reason why gRPC reflect is not working.

We will re-assess this later, but it might also be possible to use an unbuffered stream, with some minimal WAF setup. Alternatively, we plan to offer direct connectivity, which could also depending of the need solve this.

As a workaround, you should be able to spawn a debug service/worker and use grpcurl with reflection directly by accessing the gRPC service using the mesh DNS entry.

Also as mentioned before, grpcurl using the proto file is also working fine, but I agree that sometimes it’s not the simplest, especially with external dependencies.

1 Like
6

Ok, understood. I’ll support public inbound TCP connections too :slight_smile:
Thanks for all details!