Allowing for RC to 'emit' data

[rhwarrier]

The current demos that have been showcased with RC have used DB queries to pull data and generate basic metrics /dashboards. As we see an uptick in adoption, there is an urgent need for RC as infra be able to emit usable events - RC should allow adopters to generate basic telemetry, so that it can be used to compute metrics of interest.

We should initiate design discussions to arrive at some basic telemetry specs that will allow for answering of most common queries across a broad spectrum of RC use cases - no. of entires in the registry (by type etc), geographical /demographic spread, computations around VCs - numbers issued, number of users with certain kinds of VCs etc.

Requesting thoughts around collation of some core metrics that will apply across domains - once that is agreed upon, we can derive the event structures required to emit such data, and RC can subsequently provision for generation of these.

Please feel free to add thoughts

@surendrasinghs @parthlawate @SanthoshVasabhaktula @vrayulu @sukhpreetsamagra @ChakshuGautam @pramodkvarma @tejash-jl

[sankarshanmukhopadhyay]

Going over the conversations around metrics and data, having the "core" RC provide for APIs or service endpoints would make sense, which can be ingested by any well-known instance of telemetry+dashboard software and generate data required by the business.

[parthlawate]

Bringing over a related thread I had posted on the CQube community
Sunbird-cQube/community#761

[rhwarrier]

Following up on this thread:

The intent is to provide RC with the capability to ‘emit’ data for reporting purposes. What we now need to get to is
(1) what sort of data will RC need to emit?
(2) how can this data be used by an adopter?

Data generation:
Considering the basic capabilities that RC enables, we’ll start with the assumption that any metric that an RC adopter would want to track will revolve (mostly) around the 7 core Registry verbs. As a start, we will look at bringing in the ability for EC to emit events for all these actions (verbs)
There will be audit events generated for each of these actions undertaken within RC - there can be 7 sub-types for the audit events, one for each verb (Enrol, Authenticate, Consent, Attest, Claim, Discover, Update). RC will raise anonymised events that will send relevant context + some data as necessary.
We need to define a spec/ structure for the RC events - this can be based on the SB telemetry specs as well

Data consumption for adopters:
As far as an adopter is concerned, they will also need to know how the emitted can be effectively consumed
What are the different options we can enable as RC ?
Emit events into another DB table ? Emit to a Kafka topic ? Emit to an API end point that the adopter can then call?

The following are the data capabilities for RC that need to come alive:

generate relevant events that will allow for computation of metrics as necessary
create a few common datasets (basic cumulatives that could be weekly/ overall aggregates etc.) that can be pulled by adopters. (No. Of creds issued, claims made & attested etc.)

The next steps, post discussions will involve arriving at an event structure for RC

Please feel free to add & tag others as necessary. Thanks.

@pramodkvarma

@SanthoshVasabhaktula @vrayulu
@alok-os
@surendrasinghs
@parthlawate @coolbung
@tejash-jl
@sukhpreetsamagra @ChakshuGautam

[pramodkvarma]

I think we implemented a set of events from RC instance for Co-WIN and emitted via Kafka. Can @tejash-jl check with @dileepbapat and understand what was done?

[tejash-jl]

Below is the approach we are planning to incorporate. It's similar to Co-WIN implementation. We have considered that we need to aggregate data from different hosted registries. The below design can work for federated registries or for a centralised registry as well.
SunbirdRC core will emit events for every CRUD operation. Only public data + masked private fields will be sent to kafka. We will create a metrics service which consumes the data and stores it in DB (Clickhouse). The metrics service should be configured with a list of fields/columns on which aggregation needs to be done. The metrics service will expose those aggregated data via a metrics API or by creating a JSON file at the scheduled time (Ex: day-wise). The metrics API will be compliant to Prometheus. Prometheus helps to aggregate data across different registries and the same can be queried using APIs or integrating with dashboards like Grafana.

[pramodkvarma]

Given SB-RC is a low code framework, we need to allow registry fields or cred schema fields to be configured for emit (default being none emitted given we don't know the sensitivity of the fields). Each field should be configured for EMIT as below:

1. NONE - do not emit this field
2. FULL - emit the field value as-is
3. HASH - emit the field as a one-way hash (salted)
4. MASK - emit the field using a masking strategy
5. HASH-MASK - emit two values for this field, one hashed and one masked

Yes, we should telemetry emit for every CRUD API based on above config. Once the telemetry is ingested within the analytics module within the RC instance, it can be used to create metrics (to local instance) and make the metrics also publishable. Publishing to outside world (from within the RC instance) should be from the analytics module of the instance (data/metrics APIs/feeds).

Between instances, we SHOULD NOT assume any tech compatibility. If each instance publishes their metrics via APIs/feeds, others can aggregate it. We should not assume Prometheus or any other tech at all. Keep it merely data APIs/feeds in some standard JSON structures.

Makes sense? @rhwarrier @tejash-jl @parthlawate

[pramodkvarma]

Agree. I am OK with the suggestion @dileepbapat. Only one point, even internal fields (like internally generated ID etc) may be useful (in tokenized form) within the emit. So, I suggest we allow both PRIVATE and INTERNAL fields to have emit config. Makes sense?

[tejash-jl]

Yes, we can make the internal fields also configurable.

[pramodkvarma]

Yes

[parthlawate]

I like the idea of making emits configurable.. As @tejash-jl said in most cases it can be derived from the access specifiers..

However I am not clear in cases where Reporting needs some PII data to be available for a valid use case.. how will that be enabled.. Lets Say Class teacher needing to access Data of their students..

How will access to data be controlled in this case ?

[pramodkvarma]

Analytics systems trigger actions (in your example, saying, 20 students in the class needs extra support, not exactly saying which 20 students) and the trigger should be integrated back in actual transaction systems (school system, learning systems, assessment system, attendance systems, etc) for the teacher to view the full details and take action. Otherwise, analytics system will be a union of all of transaction system data.

[coolbung]

To create dashboards I can think of 2 high level approaches

1. Build dashboards based on source database (eg: in this case either Postgres or Elastic)
2. Ingest emitted data into another database (say druid, clickhouse) and build dashboards on that data

Wanted to understand the rationale behind preferring option #2.

One reason I can see is privacy, i.e. the emitted data will be designed to not have any PII. However this can be mitigated to an extent by creating "views" in the database with only the necessary fields.
Is scale a factor ?

[pramodkvarma]

Yes, given credential/registry data is sensitive, querying source data is a bad architecture approach. Anonymized stream should be emitted as events occur, which then can be aggregated into a analytics datastore for dashboards etc.

[coolbung]

Ok. So if

1. A postgres view is created that has only the select fields wrt to maintaining privacy (understand this will need to be done outside the internal / private / public field config)
2. And the postgres view is protected appropriately (separate user with read only access etc)
3. Of course as schema changes, the view may need to be updated
4. Make dashboards using a tool like metabase on top of this view
5. If needed create a slave that will be used exclusively for the reports, and use the caching capabilities of the dashboarding tool

Can this be considered a viable, production grade alternative to using emitted data to build dashboards ?

[rhwarrier]

Thank you for the discussions to arrive at an approach.
@tejash-jl and @Parth , we need to plan for the build cycles now. Suggest we share the design doc (in its current format) in this thread here, and refine it iteratively as we go along. We also need to plan for who among the community can start work on it.

The intent is to have a V1 (could be a very basic version to kick-start the emit) ready over the next two weeks, in time for the Education reference solutions.

PS: Volunteers always welcome!

[surendrasinghs]

@tejash-jl let's

1. Use Sunbird telemetry specification for defining events. Pls share it in the design
2. Keep it agnostic to Kafka and demonstrate at least two possibilities - Kafka and write to file. Let's write adapters for Kafak, RabitMQ, etc as you suggested.
3. Start with demonstrating at least one event. I understand we are already designing for generic events such as entity creation, update, etc.

Please share a design.

[varadeth]

The registry will emit events. We will have a interface through which we can send these events to various streams. i.e Kafka or RabbitMQ or API or log file
In here, we will implement kafka route. Registry will send events to kafka to which metric service will subscribe.

For the events being emitted from registry, We were thinking to send the entityId, operationType, date and the data of the entity in stringified json format to kafka topic named metrics_aggregator. Kafka will hold the data and then consumer can run on this to store this into clickhouse
operationType can be ADD, UPDATE, DELETE on the entity as and when the request is successful

Metric service, in turn will send this data to a interface to which we can configure multiple database providers.

For our use case, we will use clickhouse to aggregate the events
We evaluated clickhouse data types. It provides various data types to store the JSON values. We can store it using Nested, Tuple, JSON, Map data types
We think that JSON data type is what suits us best. All other data types have some or other complexities for insertion, retrieval and aggregation queries

Following is the example request that registry can send to topic named metrics_aggregator. We are following sunbird telemetry spec in here

{
    "eid": "ADD",
    "ets": 1678958110872,
    "ver": "3.1",
    "mid": "abc-123",    //uuid of message

    "actor": {
      "id": "123", // userId of creator or anonymous
      "type":  "User"
    },
    "context": {
      "channel": "sunbird-rc-core",
      "env": "dev"
    },
    "object": {
      "id": "xyz-123",
      "type": "Student"
    },

    "edata": {
       "identityDetails": {
           "fullName": "kevin",
           "dob": "XX-XX-XX94",
           "identityHolder": {
               "type": "XXXXport",
               "value": "XXX741"
           }
       },
       "contactDetails": {
           "email": "[email protected]",
           "mobile": "XXXXXX6789",
           "address": {
               "plot": "XX",
               "street": "XXXnue"
           }
       },
       "guardianDetails": {
           "fullName": "John",
           "relation": "Father"
       }
    }
}

clickhouse table schema can include the same. Following is the clickhouse schema to store the events of Student

create table student (
    operationType String,
    date Date,
    entityId String,
    entity JSON
)

We tried to represent it in a diagram

[rhwarrier]

Let's put together a Confluence design doc that can then be used for further comments and DC review?
cc: @SanthoshVasabhaktula @mathewjpallan

[rhwarrier]

@tejash-jl @parthlawate are there any design docs that can be shared for DC review?