Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve efficiency using redis #42

Open
t-persson opened this issue Sep 23, 2020 · 1 comment
Open

Improve efficiency using redis #42

t-persson opened this issue Sep 23, 2020 · 1 comment

Comments

@t-persson
Copy link
Collaborator

Description

Investigate the use of Redis as a cache and warm storage for the API.

The idea is that the storage component attempts to write to Redis instead of MongoDB with a TTL (remove the event after a while).
In Redis there is a LUA script or something else which will replicate this DB write over to MongoDB for cold storage.

When getting an event via the GraphQL API, request Redis. Redis will return if it exists, else it will run a LUA script or something else which will get the response from MongoDB instead (cold storage)

This will cache more common requests into warm storage, it will increase efficiency on "new" events and will keep data in cold storage for long term traceability.

Motivation

Efficiency on recent events and more common requests without the loss of long term traceability

Exemplification

When there are a lot of events, the databases will start to slow down but we don't want to clean up data as this means a loss of historical traceability.
The idea is that we have a warm storage of newly received events for immediate use in the CI/CD pipeline and a cold storage of events that are required after a long while.

Benefits

Speed and a strategy for long-term storage of events.

Possible Drawbacks

Another layer of complexity with Redis as well as locking this project to use Redis.

@magnusbaeck
Copy link
Member

A Redis cache might be a good way of improving performance when getting an event by its ID, but having Redis talk directly to MongoDB seems like an unnecessary complication. I suggest keeping MongoDB interactions entirely in eiffel-graphql-api and simply using Redis as a demand cache for event lookups.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants