Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow Configuring Cleanup Microservice for Non-Packet Logs #1745

Closed
JoshuaNeely opened this issue Dec 9, 2024 · 2 comments
Closed

Allow Configuring Cleanup Microservice for Non-Packet Logs #1745

JoshuaNeely opened this issue Dec 9, 2024 · 2 comments
Assignees
Labels
enhancement New feature or request priority Priority issues driven by customers

Comments

@JoshuaNeely
Copy link

Description
Our Minio logs bucket is filling up- specifically subdirectories not associated with packet logs.
We have tooling in plugin.txt to configure cleanup microservices based on time and size of packet logs.
We seem to lack similar functionality for limiting logs for text_logs and tool_logs, as well as NOSCOPE logs.

Minio has time-based bucket lifecycle management but these are less powerful and scalable than something based on filesize, which the cleanup microservices can do.
With bucket sizes constrained by volume mounts (like kubernetes PVCs) it is difficult to correlate time-based cleanup policies with volume size, without massively oversizing volumes.

We would like capability similar to the CMD_LOG_CYCLE_TIME, CMD_LOG_CYCLE_SIZE, to apply to all logs cosmos writes to the logs bucket.

Proposed Solution
Allow configuring the built in cleanup microservice to apply to all content in the logs bucket.
This likely needs to live somewhere outside plugin.txt. A global config file? init.sh?

Additional context
We can configure new minio cleanup policies through the cosmos-init container in the short term, and increase the size of our PVCs.

@JoshuaNeely
Copy link
Author

possibly related to this issue on script-runner log retain time
#701

@jmthomas jmthomas added the enhancement New feature or request label Dec 9, 2024
@jmthomas jmthomas added the priority Priority issues driven by customers label Jan 29, 2025
@jmthomas jmthomas self-assigned this Feb 10, 2025
@jmthomas
Copy link
Member

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request priority Priority issues driven by customers
Projects
None yet
Development

No branches or pull requests

2 participants