Agent options: allow vmodule command line flag

[noahtalerman]

Goal

User story
As a security engineer on the Agent options page in Fleet,
I want to add vmodule to the command_line_flags object
so that I can limit the amount of noise while debugging osquery issues.

Key result

Small UX improvements

Original requests

• Support the vmodule command line flag #25487

Context

• Product designer: @noahtalerman

Changes

Product

• UI changes: On the Settings > Organization settings > Agent options page, allow the user to add vmodule under command_line_flags
• CLI (fleetctl) usage changes: When using fleetctl apply -f agent-options.yml, allow the user to add vmodule under command_line_flags.
• YAML changes: In agent_options, allow the user to add vmodule under command_line_flags.
• REST API changes: In the PATCH /config API endpoint, allow the user to add vmodule under command_line_flags. Make sure the website is updated.
• Fleet's agent (fleetd) changes: No changes.
• Activity changes: No changes.
• Permissions changes: No changes.
• Changes to paid features or tiers: Fleet Free and Fleet Premium
• Transparency changes: No changes.
• First draft of test plan added
• Other reference documentation changes: No changes.
• Once shipped, requester has been notified
• Once shipped, dogfooding issue has been filed

Engineering

• Test plan is finalized
• Feature guide changes: no need
• Database schema migrations: no need
• Load testing: no need

ℹ️  Please read this issue carefully and understand it. Pay special attention to UI wireframes, especially "dev notes".

QA

• Sign off from QA: @xpkoala

Risk assessment

@xpkoala , please approve/complete the risk and test-plan sections and remove the TODOs

• Requires load testing: Yes
• Risk level: Low
• Risk description: This is built-in, though minimally documented, osquery functionality and not new tooling. We will want to be aware of this change during load testing as we should see an increase in logging.

Test plan

• In the Settings > Organization settings > Agent options page, add vmodule under command_line_flags and click the Save button. Agent options should save successfully.
  • @noahtalerman: Here's an example vmodule that we can use: https://drive.google.com/file/d/10iyU_bZ6M_6MqcQh5th7IgsC2iX2h1Wq/view
• Run fleetctl apply -f agent-options.yml, with vmodule under command_line_flags. Agent options should be updated successfully.
• Using Fleet's YAML (GitOps), in agent_options, add vmodule under command_line_flags.
• REST API changes: Using the PATCH /config API endpoint, add vmodule under command_line_flags.
  • @noahtalerman: How do we check that the vmodule was actually added? Let's chat about this during user story review.
• Check the fleetd logs to verify that the vmodule was set. Setting the flag with vmodule=init=1 should produce additional osquery logs. I do not have specific examples at this time, but a snippet of logs with and without the flag enabled can be diffed to confirm the flag is working as intended.

Testing notes

This conversation shows use-case in the wild.

Confirmation

1. Engineer: Added comment to user story confirming successful completion of test plan.
2. QA: Added comment to user story confirming successful completion of test plan.

[noahtalerman]

@dantecatalfamo what does the vmodule do? I think we want to add vmodule with a short description to the osquery_flags table.

[dantecatalfamo]

Hey @noahtalerman. My understanding is that the vmodule flag gives you fine-grain control over which osquery internal systems output logs. It's an undocumented flag and there's very little information about it outside of a small source code comment. The osquery_flags table is controlled by osquery itself and only shows non-hidden flags, so I don't think there's much we can do to make it display information about it.

[dantecatalfamo]

There are a number of other hidden flags we support that don't show up in --help or osquery_flags as well

fleet/server/fleet/agent_options.go

Lines 288 to 307 in 9e5ecfb

	// those osquery flags are not OS-specific, but are also not visible using
	// osqueryd --help or select * from osquery_flags, so they can't be generated
	// by the osquery-agent-options script.
	type OsqueryCommandLineFlagsHidden struct {
	AlsoLogToStderr bool `json:"alsologtostderr"`
	EventsStreamingPlugin string `json:"events_streaming_plugin"`
	IgnoreRegistryExceptions bool `json:"ignore_registry_exceptions"`
	LogBufSecs int32 `json:"logbufsecs"`
	LogDir string `json:"log_dir"`
	MaxLogSize int32 `json:"max_log_size"`
	MinLogLevel int32 `json:"minloglevel"`
	StopLoggingIfFullDisk bool `json:"stop_logging_if_full_disk"`
	AllowUnsafe bool `json:"allow_unsafe"`
	TLSDump bool `json:"tls_dump"`
	AuditDebug bool `json:"audit_debug"`
	AuditFIMDebug bool `json:"audit_fim_debug"`
	AuditShowPartialFIMEvents bool `json:"audit_show_partial_fim_events"`
	AuditShowUntrackedResWarnings bool `json:"audit_show_untracked_res_warnings"`
	AuditFIMShowAccesses bool `json:"audit_fim_show_accesses"`
	}

[noahtalerman]

There are a number of other hidden flags we support that don't show up in --help or osquery_flags as well

fleet/server/fleet/agent_options.go

Lines 288 to 307 in 9e5ecfb

	// those osquery flags are not OS-specific, but are also not visible using
	// osqueryd --help or select * from osquery_flags, so they can't be generated
	// by the osquery-agent-options script.
	type OsqueryCommandLineFlagsHidden struct {
	AlsoLogToStderr bool `json:"alsologtostderr"`
	EventsStreamingPlugin string `json:"events_streaming_plugin"`
	IgnoreRegistryExceptions bool `json:"ignore_registry_exceptions"`
	LogBufSecs int32 `json:"logbufsecs"`
	LogDir string `json:"log_dir"`
	MaxLogSize int32 `json:"max_log_size"`
	MinLogLevel int32 `json:"minloglevel"`
	StopLoggingIfFullDisk bool `json:"stop_logging_if_full_disk"`
	AllowUnsafe bool `json:"allow_unsafe"`
	TLSDump bool `json:"tls_dump"`
	AuditDebug bool `json:"audit_debug"`
	AuditFIMDebug bool `json:"audit_fim_debug"`
	AuditShowPartialFIMEvents bool `json:"audit_show_partial_fim_events"`
	AuditShowUntrackedResWarnings bool `json:"audit_show_untracked_res_warnings"`
	AuditFIMShowAccesses bool `json:"audit_fim_show_accesses"`
	}

@dantecatalfamo got it! I'm going to file a separate user story to expose these in osquery_flags.

This user story (#25838) is just about allowing the user to set vmodule in Fleet.

[noahtalerman]

@sharon-fdm just a reminder that this research user story is ready to spec. Can you please complete the TODOs in the "Engineering" section to get this one ready for estimation?

[sharon-fdm]

Hey team! Please add your planning poker estimate with Zenhub @dantecatalfamo @lucasmrod @sgress454

[sharon-fdm]

Please add your planning poker estimate with Zenhub @jacobshandling