Dataset Observability

[yashashkumar]

1. added default query time period to 7 days
2. user can mention time using field query_time_period in days
3. moved query services to service folder
4. renamed files
5. removed datasetId from param and added it as dataset_id

Sample response

{
    "id": "api.table.metrics",
    "ver": "v2",
    "ts": "2025-01-14T16:25:07+05:30",
    "params": {
        "status": "SUCCESS",
        "msgid": "4a7f14c3-d61e-4d4f-be78-181834eeff6d",
        "resmsgid": "8fd45785-5623-4f23-a089-4b2b5a1f3ef1"
    },
    "responseCode": "OK",
    "result": [
        {
            "category": "data_freshness",
            "status": "Healthy",
            "components": [
                {
                    "type": "average_time_difference_in_min",
                    "threshold": 5,
                    "value": 0,
                    "status": "Healthy"
                },
                {
                    "type": "freshness_query_time_in_min",
                    "threshold": 10,
                    "value": 0,
                    "status": "Healthy"
                }
            ]
        },
        {
            "category": "data_observability",
            "status": "Unhealthy",
            "components": [
                {
                    "type": "data_observability_health",
                    "status": "Unhealthy"
                },
                {
                    "type": "failure_percentage",
                    "value": 35.39823008849557
                },
                {
                    "type": "threshold_percentage",
                    "value": 5
                },
                {
                    "type": "importance_score",
                    "value": 0
                }
            ]
        },
        {
            "category": "data_volume",
            "components": [
                {
                    "type": "events_per_hour",
                    "value": 95
                },
                {
                    "type": "events_per_day",
                    "value": 765
                },
                {
                    "type": "events_per_n_day",
                    "value": 765
                },
                {
                    "type": "volume_percentage_by_hour",
                    "value": 0
                },
                {
                    "type": "volume_percentage_by_day",
                    "value": 14.17910447761194
                },
                {
                    "type": "volume_percentage_by_week",
                    "value": 0
                },
                {
                    "type": "growth_rate_percentage",
                    "value": 0
                }
            ]
        },
        {
            "category": "data_lineage",
            "components": [
                {
                    "type": "transformation_success",
                    "value": 765
                },
                {
                    "type": "dedup_success",
                    "value": 384
                },
                {
                    "type": "denormalization_success",
                    "value": 572
                },
                {
                    "type": "total_success",
                    "value": 1339
                },
                {
                    "type": "total_failed",
                    "value": 1034
                },
                {
                    "type": "transformation_failed",
                    "value": 1
                },
                {
                    "type": "dedup_failed",
                    "value": 574
                },
                {
                    "type": "denorm_failed",
                    "value": 193
                }
            ]
        },
        {
            "category": "connectors",
            "components": {
                "category": "connectors",
                "components": [
                    {
                        "id": "failed",
                        "type": "failed",
                        "value": 1608
                    },
                    {
                        "id": "api",
                        "type": "success",
                        "value": 765
                    }
                ]
            }
        },
        {
            "category": "data_quality",
            "components": [
                {
                    "type": "incidents_failed",
                    "value": 1034
                },
                {
                    "type": "incidents_success",
                    "value": 1339
                },
                {
                    "type": "total_incidents",
                    "value": 2373
                }
            ]
        }
    ]
}

To fix the problem, we need to introduce rate limiting to the Express application. The best way to do this is by using the express-rate-limit package, which allows us to set a maximum number of requests that can be made to the server within a specified time window. We will apply this rate limiter to all routes to ensure that the application is protected from excessive requests.

1. Install the express-rate-limit package.
2. Import the express-rate-limit package in the Router.ts file.
3. Set up a rate limiter with a reasonable configuration (e.g., 100 requests per 15 minutes).
4. Apply the rate limiter to all routes using app.use(limiter).

To fix the problem, we need to introduce rate limiting to the route handler on line 40. The best way to do this is by using the express-rate-limit package, which allows us to easily set up rate limiting for our Express application. We will create a rate limiter with a specified window and maximum number of requests, and then apply this rate limiter to the relevant route.

1. Install the express-rate-limit package if it is not already installed.
2. Import the express-rate-limit package in the Router.ts file.
3. Create a rate limiter with the desired configuration.
4. Apply the rate limiter to the route handler on line 40.

To fix the problem, we need to introduce a rate-limiting middleware to the Express application. The express-rate-limit package is a well-known library for this purpose. We will set up a rate limiter and apply it to the specific route handler that performs authorization.

1. Install the express-rate-limit package.
2. Import the express-rate-limit package in the file.
3. Set up a rate limiter with appropriate configuration.
4. Apply the rate limiter to the route handler that performs authorization.

To fix the problem, we need to introduce rate limiting to the dataExhaust route handler. The best way to achieve this is by using the express-rate-limit middleware. This middleware allows us to set a maximum number of requests that can be made to the route within a specified time window. We will apply this middleware to the dataExhaust route to ensure it is protected against excessive requests.

1. Install the express-rate-limit package if it is not already installed.
2. Import the express-rate-limit package in the Router.ts file.
3. Create a rate limiter with appropriate settings (e.g., maximum 100 requests per 15 minutes).
4. Apply the rate limiter to the dataExhaust route.

To fix the problem, we need to introduce rate limiting to the route handler on line 56. The best way to do this is by using the express-rate-limit package, which allows us to easily set up rate limiting for specific routes or the entire application.

1. Install the express-rate-limit package.
2. Import the express-rate-limit package in the Router.ts file.
3. Set up a rate limiter with appropriate configuration (e.g., maximum number of requests per minute).
4. Apply the rate limiter to the specific route handler on line 56.

To fix the problem, we need to introduce rate limiting to the Express application. The best way to do this is by using the express-rate-limit package, which allows us to set up rate limiting middleware easily. We will configure the rate limiter to allow a maximum of 100 requests per 15 minutes and apply it to the specific route handler that performs authorization.

1. Install the express-rate-limit package.
2. Import the express-rate-limit package in the Router.ts file.
3. Configure the rate limiter with the desired settings.
4. Apply the rate limiter to the specific route handler.

[HarishGangula]

fix the param in the urls

[HarishGangula]

param should be datasetId across all the methods

[HarishGangula]

This should be read from config or dataset config, Needs discussion

[HarishGangula]

Move this file to services folder and rename methods to get instead of handle

[HarishGangula]

The also needs discussion