Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How can I intercept RESOURCE_EXHAUSTED caused by the client sending messages that are too large? #11802

Open
SoftMemes opened this issue Jan 7, 2025 · 3 comments
Labels

Comments

@SoftMemes
Copy link

I already have a ServerInterceptor used to log and report service errors to Sentry, but this does not appear to trigger when the client sends a payload that is too large, leading to a RESOURCE_EXHAUSTED, presumably as this check is done before the call even reaches the interceptor.

I have attempted to use a ServerTransportFilter as per suggestion from the AI overlords, but while this gives me a callback, I am unable to get any details of the error.

How can I intercept these errors in code?

@kannanjgithub
Copy link
Contributor

kannanjgithub commented Jan 8, 2025

RESOURCE_EXHAUSTED is thrown in the deframer that deserializes and creates the inbound message. Deframing happens in the stream layer (NettyServerStream) and only after a successful deframing, the Call layer code such as ServerCallImpl and server observer is invoked.
Interceptor is not an appropriate means for receiving this error because interceptor is only used in the code path that starts an RPC call, and before starting to send response messages for intercepting metadata headers etc. See When to use interceptors.

@ejona86
Copy link
Member

ejona86 commented Jan 8, 2025

In general service handlers and ServerInterceptors don't see error details; they only provide errors and see cancellations. ServerStreamTracer.streamClosed() is able to see error details, typically for metrics, and I suspect it has the details here.

Since this is a client error, why do you consider it a service error? Because of the warning? Would you need to do anything if it was properly handled in gRPC and communicated back to the client as mentioned in #11245 (and thus the warning removed)? Or are you not doing streaming and thus not seeing the warning?

Our AI overlords let you down with ServerTransportFilter. It is very much unrelated, as it has nothing to do with individual RPCs.

@SoftMemes
Copy link
Author

You are right in that this is strictly a client error, however I still want alerts in Sentry on the server side when/if users start to send requests that are too large for the current configuration. In this environment, I am in full control over the service and responsible for monitoring it, but not necessarily for the client(s).

I have hacked around this now by capturing the log line directly (in my case, redirecting JUL logging to logback that we use, and intercepting the log entry there), thought it would be nice to have a cleaner way as this relies on text matching.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants