Add STDOUT logging handler for Cloud Logging integration (SCP-5902) #383
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
BACKGROUND & CHANGES
This update adds a logging handler for
STDOUT
to supplement the existing logging for ingest processes. By adding this, we will leverage the existing integration of cloud logging into Batch API jobs. This will add much greater visibility for jobs as we will not need to rely on the log file being delocalized back to the bucket. This benefits running jobs the most as we will get real-time log messages, allowing us to debug the state of a job without waiting for completion.Note: This also contains an unrelated update to the
vault-gsm-migration.sh
script that was needed to help import secrets from Vault into GSM better.MANUAL TESTING
ingest
folder:user_log.txt
: