Problem description
When an error occurs during response serialization, i.e., the responseSerialize callback of the MethodDefinition throws, this section of code extracts the error message and discards the rest of the error:
|
try { |
|
response = this.serializeMessage(message); |
|
} catch (e) { |
|
this.sendStatus({ |
|
code: Status.INTERNAL, |
|
details: `Error serializing response: ${getErrorMessage(e)}`, |
|
metadata: null, |
|
}); |
|
return; |
|
} |
Since this only ever surfaces as an INTERNAL error on the client side, the server operator has no indication in their logs that anything might be wrong until a client complains.
Additionally, even if reported by a client, the error message may be of limited use without a stack trace. In our case, I ended up needing to attach a debugger to the server process and put a breakpoint in the above catch block to log the error object incl. stack trace to the console.
I propose logging such errors or exposing some other mechanism, e.g., a callback, for library users to react to them. For example, we have integrated Sentry reporting for exactly scenarios like this, and were very confused why nothing showed up there.
Reproduction steps
Apply the following patch to the helloworld example in this repo:
diff --git a/examples/helloworld/static_codegen/greeter_server.js b/examples/helloworld/static_codegen/greeter_server.js
index ae2ab949..66df1e6a 100644
--- a/examples/helloworld/static_codegen/greeter_server.js
+++ b/examples/helloworld/static_codegen/greeter_server.js
@@ -36,6 +36,11 @@ function sayHello(call, callback) {
*/
function main() {
var server = new grpc.Server();
+
+ services.GreeterService.sayHello.responseSerialize = () => {
+ throw new Error("oops");
+ };
+
server.addService(services.GreeterService, {sayHello: sayHello});
server.bindAsync('0.0.0.0:50051', grpc.ServerCredentials.createInsecure(), (err, port) => {
if (err != null) {
Then invoke the sayHello method any way you want.
Environment
- OS name, version and architecture: Debian Bookworm AArch64
- Node version: 24.18.1
- Node installation method: Docker Hub
library/node
- Package name and version: @grpc/grpc-js@1.14.3
Additional context
No logs, which is the problem.
As for what led to this error: In our case, what happened was that bad data had ended up in a database row—a null in a PostgreSQL text[] column. This lead to generated serialization code attempting to call string(null) on a Writer object from protobufjs.
Problem description
When an error occurs during response serialization, i.e., the
responseSerializecallback of theMethodDefinitionthrows, this section of code extracts the error message and discards the rest of the error:grpc-node/packages/grpc-js/src/server-interceptors.ts
Lines 892 to 901 in 92ac80f
Since this only ever surfaces as an INTERNAL error on the client side, the server operator has no indication in their logs that anything might be wrong until a client complains.
Additionally, even if reported by a client, the error message may be of limited use without a stack trace. In our case, I ended up needing to attach a debugger to the server process and put a breakpoint in the above
catchblock to log the error object incl. stack trace to the console.I propose logging such errors or exposing some other mechanism, e.g., a callback, for library users to react to them. For example, we have integrated Sentry reporting for exactly scenarios like this, and were very confused why nothing showed up there.
Reproduction steps
Apply the following patch to the
helloworldexample in this repo:Then invoke the
sayHellomethod any way you want.Environment
library/nodeAdditional context
No logs, which is the problem.
As for what led to this error: In our case, what happened was that bad data had ended up in a database row—a
nullin a PostgreSQLtext[]column. This lead to generated serialization code attempting to callstring(null)on aWriterobject fromprotobufjs.