Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(De)Serialize message containing oneof field between protobufjs #668

Open
Joel-PeakMetrics opened this issue Jan 17, 2025 · 1 comment
Open

Comments

@Joel-PeakMetrics
Copy link

Hello,

First off, thank you for all of the work on this library. I really appreciate the ability to use typescript with protobufs (and I'm using Deno which all seems to work well).

I have a message type that has a oneof field. When serializing that message type to wire-format, I'm leveraging the Confluent Schema Registry's protobuf solution, which is using this https://www.npmjs.com/package/protobufjs. The problem is the way protobufjs converts a message's oneof field to wire-format and back again doesn't seem to work with the protobuf-ts method.

Message:

message Test {
    message InnerMessageA {
       string id = 1;
    }
    message InnerMessageB {
       uint32 id = 1;
    }
    oneof inner_field {
        InnerMessageA inner_message_a = 1;
        InnerMessageB inner_message_b = 2;
    }
}
// using protobuf-ts
const example = Test.create({
  innerField: {
    oneofKind: "innerMessageA",
    innerMessageA: {}
});
// using protobufjs
const example = Test.create({
  innerMessageA: {}
});

I've worked around the encoding problem by converting the protobuf-ts to JSON, then providing to protobufjs. However, how can I convert from protobufjs to protobuf-ts? Should I also use a JSON intermediate step?

TYIA for your help, please let me know if you need any additional info here.

@jcready
Copy link
Contributor

jcready commented Feb 4, 2025

The problem is the way protobufjs converts a message's oneof field to wire-format and back again doesn't seem to work with the protobuf-ts method.

Just to be completely clear the wire-format is binary (e.g. the input to decode() or fromBinary()), not the input to create().

The interfaces of a deserialized protobuf message are significantly different between protobuf.js and protobuf-ts. There is not going to be any support for being able to call the respective create() methods with the other one's expected input. Your best bet is to either serialize to the canonical JSON representation or the binary representation with one library and then deserialize with the other. But even that may not be perfect as protobuf.js only passes 36.5% of the required protobuf conformance tests while protobuf-ts on the other hand passes 100% of them1.

You can see all the required protobuf conformance tests that protobuf.js is failing here. I count 339 failing JSON tests and 944 failing protobuf (binary) tests. Unfortunately I'm not sure how many total tests there are of each type so it's hard to say which serialization format would yield fewer issues when doing round-trips.

Footnotes

  1. The table I linked to shows 99.9%, but that is out of date information.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants