Optimizing Log Formatting to Reduce Datadog Costs

[krisnaBukitVista]

Description

This project focuses on standardizing and formatting logs across key services to reduce the growth of log indices, optimize storage, and minimize payment costs for Datadog. By addressing unformatted logs, we aim to improve log readability, simplify troubleshooting, and lower operational expenses.

Problem

Currently, some logs are not formatted, leading to the following issues:

• Increased log indices: Unformatted logs create unnecessary log entries, inflating the number of indices.
• High costs: The growth of log indices significantly increases Datadog expenses, impacting the budget.
• Inefficiencies in debugging: Unformatted logs are harder to analyze, slowing down issue resolution.

The primary sources contributing to the issue are:

• Database service
• Hades service
• Hermes service
• Main service
• BIGRR service
• Notification service

Solution

• Log Standardization:

  • Define a unified log format (e.g., JSON structure with consistent fields such as timestamp, log_level, service_name, message, etc.).
  • Implement this format across all services.

• Configuration Updates:

  • Update logging configurations in the following services:
  • Database service
  • Hades service
  • Hermes service
  • Main service
  • BIGRR service
  • Notification service

• Log Filtering and Aggregation:

  • Remove unnecessary or redundant logs.
  • Aggregate repetitive log entries to minimize storage.

Measurement Metrics

• Log Index Reduction: Reduction in log indices within the first month.
• Cost Savings: Reduce monthly Datadog costs.
• Compliance Rate: Logs comply with the new format across all services.

SLA

• Log Index Reduction: Achieve a 25% reduction in log indices within the first month.
• Cost Savings: Reduce monthly Datadog costs by at least 20%.
• Compliance Rate: Ensure 95% of logs comply with the new format across all services.