Use normal JSON object in message envelope payload instead of base64

[cloudxxx8]

🚀 Feature Request

Relevant Package [REQUIRED]

This feature request is for all

Description [REQUIRED]

By default, Golang marshall the payload to base64 string, and it consumes much more memory

go-mod-messaging/pkg/types/message_envelope.go

Line 45 in 5dabfa0

Payload []byte `json:"payload"`

It's more efficient and easier if the payload is a normal JSON

Describe the solution you'd like

Add a new configuration and environment variable to use the message payload with normal JSON object instead of base64 string

example from the original message:

{
  "ReceivedTopic":"",
  "CorrelationID":"14a42ea6-c394-41c3-8bcd-a29b9f5e6835",
  "ApiVersion":"v2",
  "RequestID":"e6e8a2f4-eb14-4649-9e2b-175247911369",
  "ErrorCode":0,
"Payload":"eyJhcGlWZXJzaW9uIjoidjIiLCJyZXF1ZXN0SWQiOiJlNmU4YTJmNC1lYjE0LTQ2NDktOWUyYi0xNzUyNDc5MTEzNjkiLCJzdGF0dXNDb2RlIjoyMDAsImRldmljZUNvcmVDb21tYW5kIjp7ImRldmljZU5hbWUiOiJSYW5kb20tQm9vbGVhbi1EZXZpY2UiLCJwcm9maWxlTmFtZSI6IlJhbmRvbS1Cb29sZWFuLURldmljZSIsImNvcmVDb21tYW5kcyI6W3sibmFtZSI6IldyaXRlQm9vbFZhbHVlIiwic2V0Ijp0cnVlLCJwYXRoIjoiL2FwaS92Mi9kZXZpY2UvbmFtZS9SYW5kb20tQm9vbGVhbi1EZXZpY2UvV3JpdGVCb29sVmFsdWUiLCJ1cmwiOiJodHRwOi8vZWRnZXgtY29yZS1jb21tYW5kOjU5ODgyIiwicGFyYW1ldGVycyI6W3sicmVzb3VyY2VOYW1lIjoiQm9vbCIsInZhbHVlVHlwZSI6IkJvb2wifSx7InJlc291cmNlTmFtZSI6IkVuYWJsZVJhbmRvbWl6YXRpb25fQm9vbCIsInZhbHVlVHlwZSI6IkJvb2wifV19LHsibmFtZSI6IldyaXRlQm9vbEFycmF5VmFsdWUiLCJzZXQiOnRydWUsInBhdGgiOiIvYXBpL3YyL2RldmljZS9uYW1lL1JhbmRvbS1Cb29sZWFuLURldmljZS9Xcml0ZUJvb2xBcnJheVZhbHVlIiwidXJsIjoiaHR0cDovL2VkZ2V4LWNvcmUtY29tbWFuZDo1OTg4MiIsInBhcmFtZXRlcnMiOlt7InJlc291cmNlTmFtZSI6IkJvb2xBcnJheSIsInZhbHVlVHlwZSI6IkJvb2xBcnJheSJ9LHsicmVzb3VyY2VOYW1lIjoiRW5hYmxlUmFuZG9taXphdGlvbl9Cb29sQXJyYXkiLCJ2YWx1ZVR5cGUiOiJCb29sIn1dfSx7Im5hbWUiOiJCb29sIiwiZ2V0Ijp0cnVlLCJzZXQiOnRydWUsInBhdGgiOiIvYXBpL3YyL2RldmljZS9uYW1lL1JhbmRvbS1Cb29sZWFuLURldmljZS9Cb29sIiwidXJsIjoiaHR0cDovL2VkZ2V4LWNvcmUtY29tbWFuZDo1OTg4MiIsInBhcmFtZXRlcnMiOlt7InJlc291cmNlTmFtZSI6IkJvb2wiLCJ2YWx1ZVR5cGUiOiJCb29sIn1dfSx7Im5hbWUiOiJCb29sQXJyYXkiLCJnZXQiOnRydWUsInNldCI6dHJ1ZSwicGF0aCI6Ii9hcGkvdjIvZGV2aWNlL25hbWUvUmFuZG9tLUJvb2xlYW4tRGV2aWNlL0Jvb2xBcnJheSIsInVybCI6Imh0dHA6Ly9lZGdleC1jb3JlLWNvbW1hbmQ6NTk4ODIiLCJwYXJhbWV0ZXJzIjpbeyJyZXNvdXJjZU5hbWUiOiJCb29sQXJyYXkiLCJ2YWx1ZVR5cGUiOiJCb29sQXJyYXkifV19XX19",
  "ContentType":"application/json",
  "QueryParams":{}
}

propse to change to:

{
  "ReceivedTopic":"",
  "CorrelationID":"14a42ea6-c394-41c3-8bcd-a29b9f5e6835",
  "ApiVersion":"v2",
  "RequestID":"e6e8a2f4-eb14-4649-9e2b-175247911369",
  "ErrorCode":0,
"Payload":{"apiVersion":"v2","requestId":"e6e8a2f4-eb14-4649-9e2b-175247911369","statusCode":200,"deviceCoreCommand":{"deviceName":"Random-Boolean-Device","profileName":"Random-Boolean-Device","coreCommands":[{"name":"WriteBoolValue","set":true,"path":"/api/v2/device/name/Random-Boolean-Device/WriteBoolValue","url":"http://edgex-core-command:59882","parameters":[{"resourceName":"Bool","valueType":"Bool"},{"resourceName":"EnableRandomization_Bool","valueType":"Bool"}]},{"name":"WriteBoolArrayValue","set":true,"path":"/api/v2/device/name/Random-Boolean-Device/WriteBoolArrayValue","url":"http://edgex-core-command:59882","parameters":[{"resourceName":"BoolArray","valueType":"BoolArray"},{"resourceName":"EnableRandomization_BoolArray","valueType":"Bool"}]},{"name":"Bool","get":true,"set":true,"path":"/api/v2/device/name/Random-Boolean-Device/Bool","url":"http://edgex-core-command:59882","parameters":[{"resourceName":"Bool","valueType":"Bool"}]},{"name":"BoolArray","get":true,"set":true,"path":"/api/v2/device/name/Random-Boolean-Device/BoolArray","url":"http://edgex-core-command:59882","parameters":[{"resourceName":"BoolArray","valueType":"BoolArray"}]}]}},
  "ContentType":"application/json",
  "QueryParams":{}
}

[eaton-coreymutter]

Looking at the existing message envelope, it provides two things that I think we can get by other means:

• Converting non-text messages into text
  • MQTT definitely supports general binary messages
  • I don't know a lot about NATS but the client libraries support non-text messages
• Mixing and matching formats via the mime-type field
  • We can do this in the message itself via a type field
  • e.g. in another system at Eaton we have logic like "if the message starts with '{' or '[' it is JSON, otherwise the first byte will be 1 followed by a CBOR message, or the first byte will be 2 followed by a Protobuf message, etc"

so I think we can remove the envelope as is suggested in this issue.

Would we want to combine this with optional stripped-down event formats? We have some ideas around that but no fully-thought-through design as far as I know. For example, we could have the event contain only a list of {resource name, value, timestamp}, event recipients could get the device name from the topic and get resource type etc. information by looking up the device profile if needed.

[cloudxxx8]

@eaton-coreymutter I just updated the description to include a clear example.
We may not be able to remove the envelope in this release. It's a little late to make a big breaking change.
Message Envelope is used for Events, System Events, and Metrics.
EdgeX only support CBOR for Events only now.

[bcorrion]

I think it is a great idea to remove the base64 embedded message, as it makes readability very difficult without cutting and pasting into base64 decoders, and the fact that this is prevalent throughout so many APIs it is a developer's burden to use.

My primary concern is that in Cloud's example the ContentType doesn't help to distinguish what format the payload would be (requiring Corey's payload examination to guess the format). Third party clients of the API will not know what was configured at run time. So this may require another field for the encoding.

A few years ago I tried to raise the idea of using the CNCF standard CloudEvents (cloudevents.io) as they have really thought through the message/payload obstacles. If we were to make a breaking change I would just as soon adopt that standard than keep making our own path. From google: https://www.googlecloudcommunity.com/gc/Cloud-Product-Articles/Anatomy-of-a-CloudEvents-and-its-relevance-with-SAP-and-Google/ta-p/669542

[cloudxxx8]

We may use EDGEX_MSG_BASE64_PAYLOAD=true in the environment variable to determine the behavior.
We have to support the original format in some ways.

[bcorrion]

Understood - but my point is that a third party without access to the environment variable will have to deduce the payload format as Corey explained: evaluating characters to guess JSON vs base64.

[cloudxxx8]

If the third party service consume the original message envelope (base64), they should set EdgeX env EDGEX_MSG_BASE64_PAYLOAD=true to other EdgeX services.