gRpc issue between querier and query-frontend

[ankianshu3003]

Describe the bug
gRpc issue between querier and query-frontend causing closing transport issue and giving Goaway error.

Cortex version: 1.13.0-9

Querier logs

level=error ts=2022-11-28T05:46:06.182258301Z caller=frontend_processor.go:70 msg="error processing requests" address=192.168.138.46:9095 err="rpc error: code = Unavailable desc = closing transport due to: connection error: desc = \"error reading from server: EOF\", received prior goaway: code: NO_ERROR" level=error ts=2022-11-28T05:46:06.182376421Z caller=frontend_processor.go:70 msg="error processing requests" address=192.168.138.46:9095 err="rpc error: code = Unavailable desc = closing transport due to: connection error: desc = \"error reading from server: EOF\", received prior goaway: code: NO_ERROR" level=error ts=2022-11-28T05:46:06.182402418Z caller=frontend_processor.go:70 msg="error processing requests" address=192.168.138.46:9095 err="rpc error: code = Unavailable desc = closing transport due to: connection error: desc = \"error reading from server: EOF\", received prior goaway: code: NO_ERROR" level=error ts=2022-11-28T05:46:06.182429439Z caller=frontend_processor.go:70 msg="error processing requests" address=192.168.138.46:9095 err="rpc error: code = Unavailable desc = closing transport due to: connection error: desc = \"error reading from server: EOF\", received prior goaway: code: NO_ERROR

Query-frontend logs:

level=debug ts=2022-11-28T05:46:07.94667526Z caller=grpc_logging.go:53 method=/frontend.Frontend/Process duration=6m50.178493703s err="context canceled" msg="gRPC\n" level=debug ts=2022-11-28T05:46:07.947018057Z caller=grpc_logging.go:53 method=/frontend.Frontend/Process duration=6m50.1791711s err="context canceled" msg="gRPC\n" level=debug ts=2022-11-28T05:46:07.947086274Z caller=grpc_logging.go:53 method=/frontend.Frontend/Process duration=6m50.17923511s err="context canceled" msg="gRPC\n" level=debug ts=2022-11-28T05:46:07.947134714Z caller=grpc_logging.go:53 method=/frontend.Frontend/Process duration=6m50.179005985s err="context canceled" msg="gRPC\n"

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.

[friedrichg]

I have never seen "received prior goaway: code: NO_ERROR"

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.