Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add ESF specific User-Agent header in outgoing Elasticsearch requests #537

Merged
merged 11 commits into from
Nov 28, 2023
Merged

Add ESF specific User-Agent header in outgoing Elasticsearch requests #537

merged 11 commits into from
Nov 28, 2023

Conversation

girodav
Copy link
Contributor

@girodav girodav commented Nov 23, 2023 •
edited
Loading

What does this PR do?

  • Include ESF specific User-Agent header in outgoing Elasticsearch requests
  • Bump the minimum supported version for the Elastic stack to 7.17

Why is it important?

This is a common pattern with Beats, and will help with troubleshooting ESF issues in Elastic Cloud, as well as helping us to better understand customer usage of ESF within Elastic Cloud.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.md

Author's Checklist

  • [ ]

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

@github-actions GitHub Actions
Copy link

github-actions bot commented Nov 23, 2023 •
edited
Loading

File Coverage Lines Branches Missing
All files 100% 100% 100% ✅
main_aws.py 100% 100% 100% ✅
handlers/__init__.py 100% 100% 100% ✅
handlers/aws/__init__.py 100% 100% 100% ✅
handlers/aws/cloudwatch_logs_trigger.py 100% 100% 100% ✅
handlers/aws/exceptions.py 100% 100% 100% ✅
handlers/aws/handler.py 100% 100% 100% ✅
handlers/aws/kinesis_trigger.py 100% 100% 100% ✅
handlers/aws/replay_trigger.py 100% 100% 100% ✅
handlers/aws/s3_sqs_trigger.py 100% 100% 100% ✅
handlers/aws/sqs_trigger.py 100% 100% 100% ✅
handlers/aws/utils.py 100% 100% 100% ✅
share/__init__.py 100% 100% 100% ✅
share/config.py 100% 100% 100% ✅
share/environment.py 100% 100% 100% ✅
share/events.py 100% 100% 100% ✅
share/expand_event_list_from_field.py 100% 100% 100% ✅
share/factory.py 100% 100% 100% ✅
share/include_exlude.py 100% 100% 100% ✅
share/json.py 100% 100% 100% ✅
share/logger.py 100% 100% 100% ✅
share/multiline.py 100% 100% 100% ✅
share/secretsmanager.py 100% 100% 100% ✅
share/utils.py 100% 100% 100% ✅
share/version.py 100% 100% 100% ✅
shippers/__init__.py 100% 100% 100% ✅
shippers/composite.py 100% 100% 100% ✅
shippers/es.py 100% 100% 100% ✅
shippers/factory.py 100% 100% 100% ✅
shippers/logstash.py 100% 100% 100% ✅
shippers/shipper.py 100% 100% 100% ✅
storage/__init__.py 100% 100% 100% ✅
storage/decorator.py 100% 100% 100% ✅
storage/factory.py 100% 100% 100% ✅
storage/payload.py 100% 100% 100% ✅
storage/s3.py 100% 100% 100% ✅
storage/storage.py 100% 100% 100% ✅

Minimum allowed coverage is 100%

Generated by 🐒 cobertura-action against baa8f94

tommyers-elastic
tommyers-elastic reviewed Nov 23, 2023
View reviewed changes
share/utils.py Outdated Show resolved Hide resolved
tommyers-elastic
tommyers-elastic reviewed Nov 23, 2023
View reviewed changes
shippers/es.py Outdated Show resolved Hide resolved
tommyers-elastic
tommyers-elastic reviewed Nov 23, 2023
View reviewed changes
share/utils.py Outdated Show resolved Hide resolved
aspacca
aspacca reviewed Nov 23, 2023
View reviewed changes
share/version.py Outdated Show resolved Hide resolved
@girodav girodav changed the title Add user agent in ES client kwargs Add ESF specific User-Agent header in outgoing Elasticsearch requests Nov 27, 2023
@girodav girodav marked this pull request as ready for review November 27, 2023 18:07
tommyers-elastic
tommyers-elastic previously approved these changes Nov 27, 2023
View reviewed changes
Copy link

@tommyers-elastic tommyers-elastic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@tommyers-elastic tommyers-elastic self-requested a review November 28, 2023 10:17
@girodav girodav merged commit 35dc090 into main Nov 28, 2023
4 checks passed
@girodav girodav deleted the add-es-user-agent branch November 28, 2023 12:12
aspacca pushed a commit that referenced this pull request Dec 1, 2023
@girodav @dependabot
Remove delete sqs message calls and optimise json collector decorator (…
4c76f27 
…#534)

* parse the content as json only when really necessary (not single content type, or with a events list expander from field)

* make lint

* fix bugs

* bump localstack container

* add ec2 service to localstack

* fix comment

* fix typo

* reduce more complexity

* fix test for localstack 3.0.0

* fix lint

* further semplification and better performance

* remove debug printing

* remove debug printing

* clean get_by_lines unit test

* fix lint

* even simplier

* fix lint

* increase coverage

* Add ESF specific User-Agent header in outgoing Elasticsearch requests (#537)

* Add ESF specific User-Agent header in outgoing Elasticsearch requests
* Bump minimum supported Elastic stack version to 7.17

* Bump boto3 from 1.28.80 to 1.33.1 (#541)

Bumps [boto3](https://github.com/boto/boto3) from 1.28.80 to 1.33.1.
- [Release notes](https://github.com/boto/boto3/releases)
- [Changelog](https://github.com/boto/boto3/blob/develop/CHANGELOG.rst)
- [Commits](boto/boto3@1.28.80...1.33.1)

---
updated-dependencies:
- dependency-name: boto3
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump boto3 from 1.33.1 to 1.33.2 (#542)

Bumps [boto3](https://github.com/boto/boto3) from 1.33.1 to 1.33.2.
- [Release notes](https://github.com/boto/boto3/releases)
- [Changelog](https://github.com/boto/boto3/blob/develop/CHANGELOG.rst)
- [Commits](boto/boto3@1.33.1...1.33.2)

---
updated-dependencies:
- dependency-name: boto3
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* inline doc

* fix tests

* changelog

* add back empty newline in get_by_lines_parameters

* use type aliases

* add comments

* fix typo in comment

* add more comments

* add decorator unit tests

* fix decorator according to test expectations

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: Davide Girardi <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
aspacca pushed a commit that referenced this pull request Dec 6, 2023
@girodav @dependabot
Remove delete sqs message calls and optimise json collector decorator (…
abfb93d 
…#544)

* Remove sqs delete record calls

* Remove delete sqs message calls and optimise json collector decorator (#534)

* parse the content as json only when really necessary (not single content type, or with a events list expander from field)

* make lint

* fix bugs

* bump localstack container

* add ec2 service to localstack

* fix comment

* fix typo

* reduce more complexity

* fix test for localstack 3.0.0

* fix lint

* further semplification and better performance

* remove debug printing

* remove debug printing

* clean get_by_lines unit test

* fix lint

* even simplier

* fix lint

* increase coverage

* Add ESF specific User-Agent header in outgoing Elasticsearch requests (#537)

* Add ESF specific User-Agent header in outgoing Elasticsearch requests
* Bump minimum supported Elastic stack version to 7.17

* Bump boto3 from 1.28.80 to 1.33.1 (#541)

Bumps [boto3](https://github.com/boto/boto3) from 1.28.80 to 1.33.1.
- [Release notes](https://github.com/boto/boto3/releases)
- [Changelog](https://github.com/boto/boto3/blob/develop/CHANGELOG.rst)
- [Commits](boto/boto3@1.28.80...1.33.1)

---
updated-dependencies:
- dependency-name: boto3
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump boto3 from 1.33.1 to 1.33.2 (#542)

Bumps [boto3](https://github.com/boto/boto3) from 1.33.1 to 1.33.2.
- [Release notes](https://github.com/boto/boto3/releases)
- [Changelog](https://github.com/boto/boto3/blob/develop/CHANGELOG.rst)
- [Commits](boto/boto3@1.33.1...1.33.2)

---
updated-dependencies:
- dependency-name: boto3
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* inline doc

* fix tests

* changelog

* add back empty newline in get_by_lines_parameters

* use type aliases

* add comments

* fix typo in comment

* add more comments

* add decorator unit tests

* fix decorator according to test expectations

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: Davide Girardi <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Update CHANGELOG.md

* revert not updating ending offset in ExpandEventListFromField

* add typing-extensions==4.8.0 to requirements

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: girodav <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@aspacca aspacca aspacca left review comments

@tommyers-elastic tommyers-elastic tommyers-elastic approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Include ESF specific User-Agent header in outgoing Elasticsearch requests
3 participants
@girodav @aspacca @tommyers-elastic