-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Mamokari/observability #307
Open
mokarian
wants to merge
28
commits into
master
Choose a base branch
from
mamokari/observability
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+964
−200
Open
Changes from all commits
Commits
Show all changes
28 commits
Select commit
Hold shift + click to select a range
07f985a
Adds Observability Modules
5adb1bc
Fixes YML file
5bc9778
Fixes flake8 errors
cb2694a
removes python-dotenv==0.10.3 dependency
534eaec
Chages the AML_REBUILD_ENVIRONMENT to false
56a20bf
Fixes the AML image snapshot path
0f614c4
Satisfies Flake8
bb0d158
Integrates Observability onto Pipeline and one script
3a619da
Chamnges factory to register_loggers
ff513b3
Adds app_insights_connection_string to Env class
f61d444
Integrates observability with ml_service
b2d81d1
Satisfies Flake8
0904c59
Integrates diabetes_regression scripts with observability
cbd100a
Changes Observability to add AppInsightsLoggers to list of loggers on…
704d345
Adds APP_INSIGHTS_CONNECTION_STRING to logger_interface
421c14c
Removes the dependency to Env object
bbc871a
Satisfies Flake8
cc95eaf
Adds dataclasses to conda_dependencies.yml
e1a76f8
Fixes YAML indentation
07e432f
Changes log_metric to log
b48da8e
Adds more scripts to use Observability and get app_insight_connectoin_
b25c7e2
Remove AML_REBUILD_ENVIRONMENT flag
b814d47
Resolves the conflicts
13841e2
Adds diabetes_regression to scoring path
66cffef
Adds dataclasses
8328b88
Adds dataclasses to conda_dependencies_scorecopy.yml
8cac588
Adds python-dotenv and opencensus to conda_dependencies_scorecopy.yml
c113423
Merges with master
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -109,3 +109,6 @@ condaenv.* | |
.mypy_cache/ | ||
|
||
.DS_Store | ||
|
||
#pycharm | ||
.idea |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -26,7 +26,11 @@ | |
from azureml.core import Run | ||
import argparse | ||
import traceback | ||
from util.model_helper import get_model | ||
from diabetes_regression.util.model_helper import get_model | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. As @j-so mentioned, all the references to |
||
from utils.logger.logger_interface import Severity | ||
from utils.logger.observability import Observability | ||
|
||
observability = Observability() | ||
|
||
run = Run.get_context() | ||
|
||
|
@@ -42,7 +46,7 @@ | |
# load_dotenv() | ||
# sources_dir = os.environ.get("SOURCES_DIR_TRAIN") | ||
# if (sources_dir is None): | ||
# sources_dir = 'diabetes_regression' | ||
# sources_dir = '.' | ||
# path_to_util = os.path.join(".", sources_dir, "util") | ||
# sys.path.append(os.path.abspath(path_to_util)) # NOQA: E402 | ||
# from model_helper import get_model | ||
|
@@ -89,7 +93,8 @@ | |
parser.add_argument( | ||
"--allow_run_cancel", | ||
type=str, | ||
help="Set this to false to avoid evaluation step from cancelling run after an unsuccessful evaluation", # NOQA: E501 | ||
help="Set this to false to avoid evaluation step from cancelling " | ||
"run after an unsuccessful evaluation", | ||
default="true", | ||
) | ||
|
||
|
@@ -109,42 +114,47 @@ | |
tag_name = 'experiment_name' | ||
|
||
model = get_model( | ||
model_name=model_name, | ||
tag_name=tag_name, | ||
tag_value=exp.name, | ||
aml_workspace=ws) | ||
model_name=model_name, | ||
tag_name=tag_name, | ||
tag_value=exp.name, | ||
aml_workspace=ws) | ||
|
||
if (model is not None): | ||
production_model_mse = 10000 | ||
if (metric_eval in model.tags): | ||
production_model_mse = float(model.tags[metric_eval]) | ||
new_model_mse = float(run.parent.get_metrics().get(metric_eval)) | ||
if (production_model_mse is None or new_model_mse is None): | ||
print("Unable to find", metric_eval, "metrics, " | ||
"exiting evaluation") | ||
if((allow_run_cancel).lower() == 'true'): | ||
observability.log("Unable to find" + | ||
metric_eval + "metrics, exiting evaluation") | ||
if ((allow_run_cancel).lower() == 'true'): | ||
run.parent.cancel() | ||
else: | ||
print( | ||
observability.log( | ||
"Current Production model mse: {}, " | ||
"New trained model mse: {}".format( | ||
production_model_mse, new_model_mse | ||
) | ||
) | ||
|
||
if (new_model_mse < production_model_mse): | ||
print("New trained model performs better, " | ||
"thus it should be registered") | ||
observability.log("New trained model performs better, " | ||
"thus it should be registered") | ||
else: | ||
print("New trained model metric is worse than or equal to " | ||
"production model so skipping model registration.") | ||
if((allow_run_cancel).lower() == 'true'): | ||
observability.log("New trained model metric is worse " | ||
"than or equal to " | ||
"production model so skipping " | ||
"model registration.") | ||
if ((allow_run_cancel).lower() == 'true'): | ||
run.parent.cancel() | ||
else: | ||
print("This is the first model, " | ||
"thus it should be registered") | ||
observability.log("This is the first model, " | ||
"thus it should be registered") | ||
|
||
except Exception: | ||
traceback.print_exc(limit=None, file=None, chain=True) | ||
print("Something went wrong trying to evaluate. Exiting.") | ||
observability.log( | ||
description="Something went wrong trying to evaluate. Exiting.", | ||
severity=Severity.ERROR) | ||
|
||
raise |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why is this needed? This environment variable is used to keep the name of the folder, since the name will likely change during bootstrap. For this reason, we should avoid hardcoding this name in the pipeline and code.