Decision log limit question

[sirpi]

Hi,
I have the following discovery agent config:

package config
discovery = {
    "bundles": { ... },
    "decision_logs": {
        "reporting": {
            "upload_size_limit_bytes": 1048576
        },
        "service": "<service-name>"
    },
    "status": {
        "service": "<service-name>"
    }
}

So I tried to limit the decision log size limit by 1MB. Though I realized that the decision log service (which has a size limit of 2MB just to be on the safe side) started throwing errors about the payload size being exceeded. So it seems that the agent (v0.48.0) does not respect this setting for some reason. Have I configured something wrong? Or what should be the expected behaviour when the size limit is exceeded on opa side?
Thanks for your answers in advance.

[charlieegan3]

Hi Sirpi, thanks for your question. Would you be able to share the discovery configuration you're using with this. e.g.

discovery:
  resource: bundles/discovery.tar.gz
  decision: discovery/config
  ... 

[sirpi]

Hi,

The base config has this structure:

services:
- name: storage-service
  ...
- name: decision-log-service
  url: <service-url>
  headers:
    agentId: <myId>
  credentials: ...
  tls: ...
discovery:
  name: agent-config
  resource: configurations/discovery.tar.gz
  service: storage-service
  decision: config/discovery
  signing:
    keyid: global_key
keys:
  global_key:
    key: ...

I hope you find something useful here and thanks for your help.

[charlieegan3]

Hi again, I've not been able to replicate this locally. I have created a small repo here: https://github.com/charlieegan3/opa-decision-log-test. Would you be able to try that out and see if you see the 'declog service' receiving data of more than the byte limit?

I'm not sure if the repo is representative of what you're using, I am using input documents of 450kb. What size are your input documents? (I presumed smaller, but a good number to know)

I'm going to be heading out soon, I'll be back at 9am GMT tomorrow :)

[ashutosh-narkar]

@sirpi what is the exact error message you get? If the size of a decision log event exceeds the specified value for the upload_size_limit_bytes, then you'll see payload exceeded message. In that case, you'll have to bump the upload_size_limit_bytes value. But would be helpful to see the log.

[sirpi]

There are policies whose response are organized and filtered data and it can be larger than the limit of 1MB.
And the error log entry in the decision log server is the following (latest example):

2023-01-31 00:12:46.462 ERROR 1 --- [r-http-epoll-10] a.w.r.e.AbstractErrorWebExceptionHandler : ​[98628349-1785665]  500 Server Error for HTTP POST "/v1.0/api/logs"

org.springframework.core.io.buffer.DataBufferLimitException: Exceeded limit on max bytes to buffer : 2097152
at org.springframework.core.io.buffer.LimitedDataBufferList.raiseLimitException(LimitedDataBufferList.java:99) ~[spring-core-5.3.24.jar:5.3.24]

The limit mentioned here is twice as much as the one set in the opa server. Btw, what should opa do if the threshold is reached? Sending the response in chunks? Or trimming the response? Or not sending anything? I did not find the intended behaviour in the docs for this case.

And thanks @charlieegan3 for your repo, I'll check it tomorrow. Now it's getting late :)

[ashutosh-narkar]

Can you share the actual OPA log? In your case a single event could be exceeding the upload_size_limit_bytes which will result in an error. OPA will not trim the response. Some of this is documented here.

[sirpi]

Here is the OPA error log:

{"level":"error","msg":"log upload failed, server replied with HTTP 500 Internal Server Error.","plugin":"decision_logs","time":"2023-01-31T09:06:21Z"}

And yes, sometimes the decision log can be larger than the limit for a single request. Does that mean that in this case the whole decision log is sent to the decision-log-service without any control of the limit?

[ashutosh-narkar]

This seems to be an issue with the server. Here OPA is trying to upload the log but the server is not able to accept it. You'll likely need to check the server code. OPA will simply try to resend the logs.

[sirpi]

So what I would like to achieve is not getting an error if there is a single large decision log in the system.
My agent is configured to limit the decision size in 1MB.
My decision log server is configured to only accept logs smaller than 2MB.

With this configuration it is strange for me that the server gets a log which is larger than configured 2MB.
Now I understand why this happens based on the implementation.
But probably it would be more useful to replace the result in the decision log with something like { "result" : "LARGE OBJECT!" } or something like that instead of getting an error. I wouldn't like to increase the limits to avoid sending large objects through the network but it seems I cannot do that with the current implementation. And as I mentioned, some of the policies return large amount of organized and filtered data, so it is not possible to eliminate these policies from the system either.

[charlieegan3]

Using my demo repo, I changed the input document to be a larger file. I then get these logs in the OPA log stream:

Log encoding failed: upload chunk size (3400401) exceeds upload_size_limit_bytes (1048576).

OPA won't send a log which is above upload_size_limit_bytes.

With this configuration it is strange for me that the server gets a log which is larger than configured 2MB.

I'm not sure why yet, but it seems to be that something about your set up is not working since the behaviour you experience is different. Are you able to confirm that the discovery bundle is being loaded by the OPA?

{"level":"info","msg":"Discovery update processed successfully.","plugin":"discovery","time":"2023-02-01T10:18:36Z"}

[sirpi]

Thanks for your help and for the repo you created. It indeed works as intended. Now I have to figure out why my opa server behaves differently.

[charlieegan3]

Thanks for confirming sirpi :) let us know if you work it out / run into any other issues.

[sirpi]

Just before closing the issue, it was indeed a mistake on the decision log server side. This setting caused the problem:

spring:
  codec:
    max-in-memory-size: 2MB

So the 2MB was not a hard limit on the request side, but the max memory usage for JSON transformation. And that sometimes required more than 2MB of memory for a request less than 1MB.

Sorry for the inconvenience. But at least now I understand the behaviour of the "upload_size_limit_bytes" parameter, thanks for your help and clarification.

[charlieegan3]

Ahh that makes sense, JSON unmarshalled will usually be much larger. Glad you got to the bottom of it!