-
Notifications
You must be signed in to change notification settings - Fork 14.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Allow check_response to be defined in extra_options of HTTP connection #45451
Open
dabla
wants to merge
21
commits into
apache:main
Choose a base branch
from
dabla:feature/http-extra-options-check-response
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Allow check_response to be defined in extra_options of HTTP connection #45451
dabla
wants to merge
21
commits into
apache:main
from
dabla:feature/http-extra-options-check-response
+182
−161
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
2 tasks
…tch signature of HttpHook
…gHook so we don't need have to override the complex get_conn method
…of HttpHook so we don't need to override the get_conn method in DingdingHook and LivyHook
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR allows you to define the check_response option in the extra options fields of the HTTP connection so that you can disable the check globally for that connection instead of always having to pass it to each HttpOperator using that same connection id.
I also took the opportunity to refactor the get_conn method of the HttpHook so you can easily override the default host and default headers in the specialized hooks so those subclasses (e.g. LivyHook or DingdingHook) don't need to override the get_conn method anymore, otherwise as the signature has changed (e.g. added extra_options parameter) those has to be changed also and the exisiting solution wasn't DRY also, now it is.
In the past if you want to disable the response_check in the HttpHook through the HtpOperator, you had to define the HttpOperator like this:
Now thanks to this PR, you can define the check_response globally in the extra_options of the connection, so it will be applied to each HttpOperator using that connection.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rst
or{issue_number}.significant.rst
, in newsfragments.