Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Standardise loggers in python lambda functions #645

Closed
wants to merge 1 commit into from
Closed

Standardise loggers in python lambda functions #645

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions ...d/components/icav2-copy-files/check_or_launch_job_lambda_py/check_or_launch_job_lambda.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,8 +50,12 @@
from mypy_boto3_secretsmanager import SecretsManagerClient

# Set logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
logging.basicConfig(
level=logging.INFO,
force=True,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will remove any preconfigured handlers...
Is that what we want?

The accepted answer claims that the default AWS handler "might also add some metadata to the record if available", which sounds like it could be useful?

Happy to accept the change. Longer term we should probably switch to the lambda power tools...

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh those docs are actually very handy!

Hmm, yes you might be right, from this below, the logger may in fact set the requestId as well.

Using structured JSON logs with Python
If you select JSON for your function’s log format, Lambda will send logs output by the Python standard logging library to CloudWatch as structured JSON. Each JSON log object contains at least four key value pairs with the following keys:

"timestamp" - the time the log message was generated

"level" - the log level assigned to the message

"message" - the contents of the log message

"requestId" - the unique request ID for the function invocation

We could rearrange from

import logging
logging.basicConfig(
    level=logging.INFO,
    force=True,
    format='%(asctime)s %(message)s'
)
logger = logging.getLogger()

To just

import logging
logger = logging.getLogger()
logger.setLevel(logging.INFO)

And then we still get the same log levels, just lose the formatting a bit

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If that works locally...
I think the issue was that then on the local env there may not be a log handler at all, hence the fallback to the basisConfig if no handler was found.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the issue was that then on the local env there may not be a log handler at all,

Yes, the answer that looks for handlers didn't seem very fun.

Something worth considering is this too - https://docs.powertools.aws.dev/lambda/python/latest/core/logger/

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, the power tools logging support. That's what I meant earlier.
(but there are also other reasons to use the power tools)

format='%(asctime)s %(message)s'
)
logger = logging.getLogger()

# Globals
SUCCESS_STATES = [
Expand Down
6 changes: 5 additions & 1 deletion ...tq-list-rows-to-cwl-input/fastq_list_rows_to_cwl_input_py/fastq_list_rows_to_cwl_input.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,12 @@
from typing import Dict, Union

# Set logger
logging.basicConfig(
level=logging.INFO,
force=True,
format='%(asctime)s %(message)s'
)
logger = logging.getLogger()
logger.setLevel(logging.INFO)


def pascal_case_to_snake_case(pascal_case_str: str) -> str:
Expand Down
6 changes: 5 additions & 1 deletion ...lesheet/get_metadata_objects_from_samplesheet_py/get_metadata_objects_from_samplesheet.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,12 @@
from metadata_tools import get_all_libraries

# Logger
logging.basicConfig(
level=logging.INFO,
force=True,
format='%(asctime)s %(message)s'
)
logger = logging.getLogger()
logger.setLevel(logging.INFO)


def get_library_objs(library_id_list: List[str]) -> List[Dict]:
Expand Down
6 changes: 5 additions & 1 deletion ...icav2-ready-event-handler/icav2_launch_pipeline_lambda_py/icav2_launch_pipeline_lambda.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -154,8 +154,12 @@
ICAV2_BASE_URL = "https://ica.illumina.com/ica/rest"

# Set loggers
logging.basicConfig(
level=logging.INFO,
force=True,
format='%(asctime)s %(message)s'
)
logger = logging.getLogger()
logger.setLevel(logging.INFO)


def get_secrets_manager_client() -> 'SecretsManagerClient':
Expand Down
8 changes: 6 additions & 2 deletions ...load/stateless/stacks/cttso-v2-pipeline-manager/lambdas/check_success_py/check_success.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,8 +28,12 @@
from wrapica.enums import DataType

# Set logger
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
logging.basicConfig(
level=logging.INFO,
force=True,
format='%(asctime)s %(message)s'
)
logger = logging.getLogger()

if typing.TYPE_CHECKING:
from mypy_boto3_secretsmanager.client import SecretsManagerClient
Expand Down
10 changes: 7 additions & 3 deletions ...ateless/stacks/cttso-v2-pipeline-manager/lambdas/compress_icav2_vcf/compress_icav2_vcf.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,9 +25,13 @@
from mypy_boto3_ssm import SSMClient
from mypy_boto3_secretsmanager import SecretsManagerClient

logger = logging.getLogger(__name__)

logging.basicConfig(level=logging.INFO)
# Set logger
logging.basicConfig(
level=logging.INFO,
force=True,
format='%(asctime)s %(message)s'
)
logger = logging.getLogger()

ICAV2_BASE_URL = "https://ica.illumina.com/ica/rest"

Expand Down
6 changes: 5 additions & 1 deletion ...tateless/stacks/cttso-v2-pipeline-manager/lambdas/delete_cache_uri_py/delete_cache_uri.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,8 +33,12 @@
ICAV2_BASE_URL = "https://ica.illumina.com/ica/rest"

# Set loggers
logging.basicConfig(
level=logging.INFO,
force=True,
format='%(asctime)s %(message)s'
)
logger = logging.getLogger()
logger.setLevel(logging.INFO)


def get_secrets_manager_client() -> 'SecretsManagerClient':
Expand Down
6 changes: 5 additions & 1 deletion ...v2-pipeline-manager/lambdas/generate_copy_manifest_dict_py/generate_copy_manifest_dict.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,8 +50,12 @@
ICAV2_BASE_URL = "https://ica.illumina.com/ica/rest"

# Set loggers
logging.basicConfig(
level=logging.INFO,
force=True,
format='%(asctime)s %(message)s'
)
logger = logging.getLogger()
logger.setLevel(logging.INFO)


def get_secrets_manager_client() -> 'SecretsManagerClient':
Expand Down
6 changes: 5 additions & 1 deletion ...tateless/stacks/cttso-v2-pipeline-manager/lambdas/set_outputs_json_py/set_outputs_json.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,8 +46,12 @@
ICAV2_BASE_URL = "https://ica.illumina.com/ica/rest"

# Set loggers
logging.basicConfig(
level=logging.INFO,
force=True,
format='%(asctime)s %(message)s'
)
logger = logging.getLogger()
logger.setLevel(logging.INFO)


def get_secrets_manager_client() -> 'SecretsManagerClient':
Expand Down
8 changes: 6 additions & 2 deletions ...andx-pipeline-manager/lambdas/generate_pieriandx_objects_py/generate_pieriandx_objects.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,8 +102,12 @@
]

# Set basic logger
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
logging.basicConfig(
level=logging.INFO,
force=True,
format='%(asctime)s %(message)s'
)
logger = logging.getLogger()



Expand Down
7 changes: 5 additions & 2 deletions ...stacks/pieriandx-pipeline-manager/lambdas/generate_samplesheet_py/generate_samplesheet.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,9 +31,12 @@
ICAV2_BASE_URL = "https://ica.illumina.com/ica/rest"

# Set loggers
logging.basicConfig(
level=logging.INFO,
force=True,
format='%(asctime)s %(message)s'
)
logger = logging.getLogger()
logger.setLevel(logging.INFO)


def get_secrets_manager_client() -> 'SecretsManagerClient':
"""
Expand Down
8 changes: 6 additions & 2 deletions ...r/lambdas/get_informaticsjob_and_report_status_py/get_informaticsjob_and_report_status.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,8 +38,12 @@
from pieriandx_pipeline_tools.utils.secretsmanager_helpers import set_pieriandx_env_vars

# Set logger
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
logging.basicConfig(
level=logging.INFO,
force=True,
format='%(asctime)s %(message)s'
)
logger = logging.getLogger()


JOB_STATUS_BOOL = {
Expand Down
6 changes: 5 additions & 1 deletion ..._2/fastq-list-rows-event-shower/lambdas/get_demultiplex_stats_py/get_demultiplex_stats.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,8 +56,12 @@
ICAV2_BASE_URL = "https://ica.illumina.com/ica/rest"

# Set loggers
logging.basicConfig(
level=logging.INFO,
force=True,
format='%(asctime)s %(message)s'
)
logger = logging.getLogger()
logger.setLevel(logging.INFO)


def get_secrets_manager_client() -> 'SecretsManagerClient':
Expand Down
7 changes: 5 additions & 2 deletions ...onstructs/clag/part_2/fastq-list-rows-event-shower/lambdas/get_fastqc_stats/run_fastqc.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,9 +64,12 @@
"""

# Set loggers
logging.basicConfig(
level=logging.INFO,
force=True,
format='%(asctime)s %(message)s'
)
logger = logging.getLogger()
logger.setLevel(logging.INFO)


def get_secrets_manager_client() -> 'SecretsManagerClient':
"""
Expand Down
6 changes: 5 additions & 1 deletion ...s/clag/part_2/fastq-list-rows-event-shower/lambdas/get_sequali_stats/get_sequali_stats.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,8 +75,12 @@
"""

# Set loggers
logging.basicConfig(
level=logging.INFO,
force=True,
format='%(asctime)s %(message)s'
)
logger = logging.getLogger()
logger.setLevel(logging.INFO)


def get_secrets_manager_client() -> 'SecretsManagerClient':
Expand Down
8 changes: 6 additions & 2 deletions ...plete-to-cttsov2-ready/lambdas/build_cttsov2_samplesheet_py/build_cttso_v2_samplesheet.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,12 @@
from typing import Optional
import logging

logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)
logging.basicConfig(
level=logging.INFO,
force=True,
format='%(asctime)s %(message)s'
)
logger = logging.getLogger()

# Globals
HEADER = {
Expand Down
6 changes: 5 additions & 1 deletion ...ect_qc_metrics_from_alignment_directory_py/collect_qc_metrics_from_alignment_directory.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,8 +59,12 @@
ICAV2_BASE_URL = "https://ica.illumina.com/ica/rest"

# Set logger
logging.basicConfig(
level=logging.INFO,
force=True,
format='%(asctime)s %(message)s'
)
logger = logging.getLogger()
logger.setLevel(logging.INFO)


def get_secrets_manager_client() -> 'SecretsManagerClient':
Expand Down
6 changes: 5 additions & 1 deletion ...w-qc-complete-event/lambdas/generate_event_data_objects_py/generate_event_data_objects.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,8 +28,12 @@
from typing import Dict

# Set log level
logging.basicConfig(
level=logging.INFO,
force=True,
format='%(asctime)s %(message)s'
)
logger = logging.getLogger()
logger.setLevel(logging.INFO)


def handler(event, context) -> Dict[str, Dict[str, str]]:
Expand Down
7 changes: 5 additions & 2 deletions ...2-output-to-pieriandx-ready-event/lambdas/get_data_from_redcap_py/get_data_from_redcap.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,9 +24,12 @@
from mypy_boto3_lambda import LambdaClient

# Set logger
logging.basicConfig(level=logging.INFO)
logging.basicConfig(
level=logging.INFO,
force=True,
format='%(asctime)s %(message)s'
)
logger = logging.getLogger()
logger.setLevel(level=logging.INFO)

# Globals
AUS_TIMEZONE = pytz.timezone("Australia/Melbourne")
Expand Down
8 changes: 6 additions & 2 deletions ...x-ready-event/lambdas/get_deidentified_case_metadata_py/get_deidentified_case_metadata.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,8 +36,12 @@


# Set logger
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
logging.basicConfig(
level=logging.INFO,
force=True,
format='%(asctime)s %(message)s'
)
logger = logging.getLogger()

# Globals
AUS_TIMEZONE = pytz.timezone("Australia/Melbourne")
Expand Down
8 changes: 6 additions & 2 deletions ...iandx-ready-event/lambdas/get_identified_case_metadata_py/get_identified_case_metadata.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,8 +49,12 @@
import logging

# Set logger
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
logging.basicConfig(
level=logging.INFO,
force=True,
format='%(asctime)s %(message)s'
)
logger = logging.getLogger()

# Globals
AUS_TIMEZONE = pytz.timezone("Australia/Melbourne")
Expand Down
7 changes: 5 additions & 2 deletions ...-to-pieriandx-ready-event/lambdas/get_pieriandx_data_files_py/get_pieriandx_data_files.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,9 +52,12 @@
}

# Set loggers
logging.basicConfig(
level=logging.INFO,
force=True,
format='%(asctime)s %(message)s'
)
logger = logging.getLogger()
logger.setLevel(logging.INFO)


def get_secrets_manager_client() -> 'SecretsManagerClient':
"""
Expand Down
6 changes: 5 additions & 1 deletion ...ess/stacks/transcriptome-pipeline-manager/lambdas/set_outputs_json_py/set_outputs_json.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,8 +44,12 @@
ICAV2_BASE_URL = "https://ica.illumina.com/ica/rest"

# Set logger
logging.basicConfig(
level=logging.INFO,
force=True,
format='%(asctime)s %(message)s'
)
logger = logging.getLogger()
logger.setLevel(logging.INFO)


def get_secrets_manager_client() -> 'SecretsManagerClient':
Expand Down
6 changes: 5 additions & 1 deletion ...less/stacks/tumor-normal-pipeline-manager/lambdas/set_outputs_json_py/set_outputs_json.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,8 +45,12 @@
ICAV2_BASE_URL = "https://ica.illumina.com/ica/rest"

# Set logger
logging.basicConfig(
level=logging.INFO,
force=True,
format='%(asctime)s %(message)s'
)
logger = logging.getLogger()
logger.setLevel(logging.INFO)


def get_secrets_manager_client() -> 'SecretsManagerClient':
Expand Down
6 changes: 5 additions & 1 deletion ...stacks/wgts-alignment-qc-pipeline-manager/lambdas/set_outputs_json_py/set_outputs_json.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,8 +48,12 @@


# Set logger
logging.basicConfig(
level=logging.INFO,
force=True,
format='%(asctime)s %(message)s'
)
logger = logging.getLogger()
logger.setLevel(logging.INFO)


def get_secrets_manager_client() -> 'SecretsManagerClient':
Expand Down