Configurable default log_level and log_format for injected Vault Agent containers

[itspngu]

Is your feature request related to a problem? Please describe.
While it is possible to configure the log level & format for the injector deployment itself using the command line or environment variables, and configuring the log level & format for injected Vault Agent sidecars on a per-pod basis using annotations, there is no way to configure a default log level & format for injected Vault Agent sidecars globally (e.g. setting the log format of all injected Vault Agent containers to json instead of default unless explicitly specified otherwise in the per-pod annotation).

Describe the solution you'd like
Add a means to override the defaults for injected Vault Agent containers' log level and format instead of hardcoding them to info and default.

Describe alternatives you've considered
Using the per-pod annotations to configure these settings is, albeit a working solution, getting cumbersome once you have a lot of workloads relying on Vault Agent. Adding vault.hashicorp.com/log-format: "json" yields is supposed to yield the needed result of getting JSON log output from the Agent containers, but having to do so in dozens of places (regardless of whether it's manually or through automation) instead of being able to configure the injector accordingly feels like a missing feature.

Additional context
If I understand the code correctly, these 2 lines are the culprit: https://github.com/hashicorp/vault-k8s/blob/main/agent-inject/agent/agent.go#L378-L379

Edited: Only the vault.hashicorp.com/log-level annotation works, vault.hashicorp.com/log-format appears to have no effect, so this is actually a bug report on top of a feature request.

[itspngu]

Actually, the annotation isn't working either - when vault.hashicorp.com/log-format: "json" is added to a pod template, the injected Vault Agent container still logs in the default format.

[itspngu]

/kind bug

[tvoran]

Hi @itspngu, when I set vault.hashicorp.com/log-format: "json" I do see json logging come from agent, though there are some non-json log lines (especially at the debug level) which is maybe what you're seeing? They look like they're coming from consul-template, which agent uses under the hood.

Regardless, you should open an issue on the vault repo for issues with agent logging format. There's been some discussion over there about it recently: hashicorp/vault#18081

[tvoran]

Being able to set the default log level and format for injected agents would be good to have. I think we'd probably want to add a couple new flags and env variables, probably something like agent-log-level and agent-log-format here:

vault-k8s/subcommand/injector/flags.go

Line 136 in 5827448

func (c *Command) init() {

The names of the environment variables for the setting gets a little more complicated, since there are already AGENT_INJECT_LOG_LEVEL and AGENT_INJECT_LOG_FORMAT for the injector service itself.

[itspngu]

Hi @itspngu, when I set vault.hashicorp.com/log-format: "json" I do see json logging come from agent, though there are some non-json log lines (especially at the debug level) which is maybe what you're seeing? They look like they're coming from consul-template, which agent uses under the hood.

Regardless, you should open an issue on the vault repo for issues with agent logging format. There's been some discussion over there about it recently: hashicorp/vault#18081

You're right, I didn't look at it for long enough to realize that it's only messages of DEBUG and TRACE severity levels which unconditionally log in text format, rather than json. That'd indeed be something to report in the main Vault repository, however I'd like to focus on the original intent of this issue for now.

If we can find names for the proposed environment variables (or skip them and just opt for CLI flags), I'd happily implement this and put it up for review. If I were to be nitpicky, I'd say that the AGENT_INJECT_LOG_LEVEL and AGENT_INJECT_LOG_FORMAT variable names are misleading, and could be changed to AGENT_INJECTOR_LOG_LEVEL and AGENT_INJECTOR_LOG_FORMAT, freeing the other ones for configuring the agent itself, but that would break existing configurations.

[Kyslik]

We experience this very issue and it is a bit annoying as it produces a lot of logs that have no value. For clarity, in the end we want to use log-level: error, below example is just to show that some components of agent don't really respect the log-format configuration.

In our case I configured the agent log level and log format using annotations on the workload being injected:

spec.template.metadata.annotations:
  vault.hashicorp.com/log-format: json
  vault.hashicorp.com/log-level: debug

and this is the snipped output:

{"@level":"info","@message":"creating file sink","@module":"sink.file","@timestamp":"2023-07-13T10:16:55.917053Z"}
{"@level":"info","@message":"file sink configured","@module":"sink.file","@timestamp":"2023-07-13T10:16:55.917178Z","mode":416,"path":"/home/vault/.vault-token"}
{"@level":"info","@message":"starting template server","@module":"template.server","@timestamp":"2023-07-13T10:16:55.917410Z"}
2023-07-13T10:16:55.917Z [INFO] (runner) creating new runner (dry: false, once: false)
2023-07-13T10:16:55.917Z [DEBUG] (runner) final config: {SNIPPED JSON}
2023-07-13T10:16:55.918Z [INFO] (runner) creating watcher
==> Vault agent started! Log data will stream in below:

==> Vault agent configuration:

                     Cgo: disabled
               Log Level: debug
                 Version: Vault v1.11.2, built 2022-07-29T09:48:47Z
             Version Sha: 3a8aa12eba357ed2de3192b15c99c717afdeb2b5

{"@level":"info","@message":"starting auth handler","@module":"auth.handler","@timestamp":"2023-07-13T10:16:55.918919Z"}
// SNIP "json" logs
{"@level":"info","@message":"template server received new token","@module":"template.server","@timestamp":"2023-07-13T10:16:56.035314Z"}
2023-07-13T10:16:56.035Z [INFO] (runner) stopping
// SNIP "standard" logs
2023-07-13T10:16:56.143Z [DEBUG] (runner) all templates rendered

I would expect everything to be in JSON, including the ==> Vault agent started! Log data will stream in below:.

[eg-chdev]

Any updates on this issue or if it was consumed as a fix in another deployment or release?