You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Investigate the use of Redis as a cache and warm storage for the API.
The idea is that the storage component attempts to write to Redis instead of MongoDB with a TTL (remove the event after a while).
In Redis there is a LUA script or something else which will replicate this DB write over to MongoDB for cold storage.
When getting an event via the GraphQL API, request Redis. Redis will return if it exists, else it will run a LUA script or something else which will get the response from MongoDB instead (cold storage)
This will cache more common requests into warm storage, it will increase efficiency on "new" events and will keep data in cold storage for long term traceability.
Motivation
Efficiency on recent events and more common requests without the loss of long term traceability
Exemplification
When there are a lot of events, the databases will start to slow down but we don't want to clean up data as this means a loss of historical traceability.
The idea is that we have a warm storage of newly received events for immediate use in the CI/CD pipeline and a cold storage of events that are required after a long while.
Benefits
Speed and a strategy for long-term storage of events.
Possible Drawbacks
Another layer of complexity with Redis as well as locking this project to use Redis.
The text was updated successfully, but these errors were encountered:
A Redis cache might be a good way of improving performance when getting an event by its ID, but having Redis talk directly to MongoDB seems like an unnecessary complication. I suggest keeping MongoDB interactions entirely in eiffel-graphql-api and simply using Redis as a demand cache for event lookups.
Description
Investigate the use of Redis as a cache and warm storage for the API.
The idea is that the storage component attempts to write to Redis instead of MongoDB with a TTL (remove the event after a while).
In Redis there is a LUA script or something else which will replicate this DB write over to MongoDB for cold storage.
When getting an event via the GraphQL API, request Redis. Redis will return if it exists, else it will run a LUA script or something else which will get the response from MongoDB instead (cold storage)
This will cache more common requests into warm storage, it will increase efficiency on "new" events and will keep data in cold storage for long term traceability.
Motivation
Efficiency on recent events and more common requests without the loss of long term traceability
Exemplification
When there are a lot of events, the databases will start to slow down but we don't want to clean up data as this means a loss of historical traceability.
The idea is that we have a warm storage of newly received events for immediate use in the CI/CD pipeline and a cold storage of events that are required after a long while.
Benefits
Speed and a strategy for long-term storage of events.
Possible Drawbacks
Another layer of complexity with Redis as well as locking this project to use Redis.
The text was updated successfully, but these errors were encountered: