Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Datasource processes exit with the mapget process #17

Open
l-laura opened this issue Jul 19, 2023 · 0 comments
Open

Datasource processes exit with the mapget process #17

l-laura opened this issue Jul 19, 2023 · 0 comments

Comments

@l-laura
Copy link
Contributor

l-laura commented Jul 19, 2023

When the mapget process is killed with the SIGKILL, the datasource processes spun up by it continue running.

There should be a POST /heartbeat request from mapget to datasources. Datasource subprocesses read the MAPGET_DATASOURCE_PROCESS_TIMEOUT environment variable, which indicates the timeout window in seconds. If this variable exists, the process shuts down when a heartbeat has not been received from mapget in the specified time window.

We should add a --datasource-executable-timeout command line argument to mapget.

The datasources should reply to the /heartbeat request with their unique datasource nodeID. In the future, mapget should use the keep-alive reply to:

  1. Check if a datasource process is still alive. If not, mapget reports it and removes worker threads for the datasource.
  2. Validate that the datasource at the given endpoint has the same ID as the originally registered datasource.
@josephbirkner josephbirkner added this to the 0.2.0 milestone Jul 20, 2023
@josephbirkner josephbirkner removed this from the 0.2.0 milestone Sep 14, 2023
@MisterGC MisterGC added this to the 0.4.0 milestone Nov 13, 2023
@MisterGC MisterGC removed this from the 0.4.0 milestone Nov 15, 2023
@l-laura l-laura removed their assignment Dec 21, 2023
@MisterGC MisterGC modified the milestone: 0.5.0 Jan 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants