Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add delimited payload token filter #7978 #7988

Conversation

AntonEliatra
Copy link
Contributor

Description

Add delimited payload token filter docs

Issues Resolved

Closes #7978

Version

all

Checklist

  • By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and subject to the Developers Certificate of Origin.
    For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Copy link

Thank you for submitting your PR. The PR states are In progress (or Draft) -> Tech review -> Doc review -> Editorial review -> Merged.

Before you submit your PR for doc review, make sure the content is technically accurate. If you need help finding a tech reviewer, tag a maintainer.

When you're ready for doc review, tag the assignee of this PR. The doc reviewer may push edits to the PR directly or leave comments and editorial suggestions for you to address (let us know in a comment if you have a preference). The doc reviewer will arrange for an editorial review.

@kolchfa-aws
Copy link
Collaborator

@udabhas Could you review this PR for technical accuracy?

@kolchfa-aws kolchfa-aws added analyzers 3 - Tech review PR: Tech review in progress labels Aug 15, 2024
@kolchfa-aws kolchfa-aws assigned vagimeli and unassigned kolchfa-aws Aug 15, 2024
@vagimeli
Copy link
Contributor

@varun-lodaya The documentation is awaiting tech review and approval, which is delaying progress. Could you please suggest alternative reviewers who can assist with this task in a timely manner? We're eager to move this forward. Thank you.

@vagimeli vagimeli added the Needs SME Waiting on input from subject matter expert label Aug 29, 2024
Signed-off-by: AntonEliatra <[email protected]>
@vagimeli
Copy link
Contributor

vagimeli commented Oct 3, 2024

@varun-lodaya The documentation is awaiting tech review and approval, which is delaying progress. Could you please suggest alternative reviewers who can assist with this task in a timely manner? We're eager to move this forward. Thank you.

@varun-lodaya This is over a month old. We need tech review approval to move it forward in the documentation process. Please review this week or provide a peer who can review it. Thank you.

Signed-off-by: Anton Rubin <[email protected]>
Copy link
Collaborator

@kolchfa-aws kolchfa-aws left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you, @AntonEliatra! Please see my changes and let me know if you have any questions.

_analyzers/token-filters/delimited-payload.md Outdated Show resolved Hide resolved
_analyzers/token-filters/delimited-payload.md Outdated Show resolved Hide resolved
_analyzers/token-filters/delimited-payload.md Outdated Show resolved Hide resolved
_analyzers/token-filters/delimited-payload.md Outdated Show resolved Hide resolved
_analyzers/token-filters/delimited-payload.md Outdated Show resolved Hide resolved
_analyzers/token-filters/delimited-payload.md Outdated Show resolved Hide resolved
_analyzers/token-filters/delimited-payload.md Outdated Show resolved Hide resolved
_analyzers/token-filters/delimited-payload.md Outdated Show resolved Hide resolved
_analyzers/token-filters/delimited-payload.md Outdated Show resolved Hide resolved
@kolchfa-aws kolchfa-aws assigned kolchfa-aws and unassigned vagimeli Oct 31, 2024
AntonEliatra and others added 2 commits November 1, 2024 11:02
Co-authored-by: kolchfa-aws <[email protected]>
Signed-off-by: AntonEliatra <[email protected]>
Signed-off-by: AntonEliatra <[email protected]>
Copy link
Collaborator

@kolchfa-aws kolchfa-aws left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you, @AntonEliatra! LGTM

Copy link
Collaborator

@natebower natebower left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@AntonEliatra @kolchfa-aws Please see my comments and changes and let me know if you have any questions. Thanks!

_analyzers/token-filters/delimited-payload.md Outdated Show resolved Hide resolved
# Delimited payload token filter

The `delimited_payload` token filter is used to parse tokens containing payloads during the analysis process. For example, the string `red|1.5 fast|2.0 car|1.0` is parsed into tokens `red` (with a payload of `1.5`), `fast` (with a payload of `2.0), and `car` (with a payload of `1.0`). This is particularly useful when your tokens have additional associated data (like weights, scores, or other numeric values) that you can use for scoring or custom query logic. The filter can handle different types of payloads, including integer, float, and strings. and attach payloads (extra metadata) to tokens

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Above: Confirm that my changes to the last sentence retain technical accuracy.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This still makes sense

_analyzers/token-filters/delimited-payload.md Outdated Show resolved Hide resolved
_analyzers/token-filters/delimited-payload.md Outdated Show resolved Hide resolved
_analyzers/token-filters/delimited-payload.md Outdated Show resolved Hide resolved
_analyzers/token-filters/delimited-payload.md Outdated Show resolved Hide resolved
_analyzers/token-filters/index.md Outdated Show resolved Hide resolved
@kolchfa-aws kolchfa-aws merged commit 63d0706 into opensearch-project:main Nov 5, 2024
5 checks passed
@kolchfa-aws kolchfa-aws added the backport 2.17 Backport for version 2.17 label Nov 5, 2024
@opensearch-trigger-bot
Copy link
Contributor

The backport to 2.17 failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-2.17 2.17
# Navigate to the new working tree
pushd ../.worktrees/backport-2.17
# Create a new branch
git switch --create backport/backport-7988-to-2.17
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 63d07069c4940b0f58f6125bfc8e48b9c89aa532
# Push it to GitHub
git push --set-upstream origin backport/backport-7988-to-2.17
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-2.17

Then, create a pull request where the base branch is 2.17 and the compare/head branch is backport/backport-7988-to-2.17.

@kolchfa-aws kolchfa-aws added backport 2.17 Backport for version 2.17 and removed backport 2.17 Backport for version 2.17 labels Nov 5, 2024
opensearch-trigger-bot bot pushed a commit that referenced this pull request Nov 5, 2024
* Adding first draft of delimited-payload token filter WIP

Signed-off-by: Anton Rubin <[email protected]>

* adding the rest of the details to delimited payload token filter

Signed-off-by: Anton Rubin <[email protected]>

* Update delimited-payload.md

Signed-off-by: AntonEliatra <[email protected]>

* updating parameter table

Signed-off-by: Anton Rubin <[email protected]>

* Apply suggestions from code review

Co-authored-by: kolchfa-aws <[email protected]>
Signed-off-by: AntonEliatra <[email protected]>

* Update delimited-payload.md

Signed-off-by: AntonEliatra <[email protected]>

* Apply suggestions from code review

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: AntonEliatra <[email protected]>

* fixing broken link

Signed-off-by: Anton Rubin <[email protected]>

---------

Signed-off-by: Anton Rubin <[email protected]>
Signed-off-by: AntonEliatra <[email protected]>
Co-authored-by: kolchfa-aws <[email protected]>
Co-authored-by: Nathan Bower <[email protected]>
(cherry picked from commit 63d0706)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
@kolchfa-aws kolchfa-aws added backport 2.17 Backport for version 2.17 and removed backport 2.17 Backport for version 2.17 labels Nov 5, 2024
@opensearch-trigger-bot
Copy link
Contributor

The backport to 2.17 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-2.17 2.17
# Navigate to the new working tree
pushd ../.worktrees/backport-2.17
# Create a new branch
git switch --create backport/backport-7988-to-2.17
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 63d07069c4940b0f58f6125bfc8e48b9c89aa532
# Push it to GitHub
git push --set-upstream origin backport/backport-7988-to-2.17
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-2.17

Then, create a pull request where the base branch is 2.17 and the compare/head branch is backport/backport-7988-to-2.17.

epugh pushed a commit to o19s/documentation-website that referenced this pull request Nov 23, 2024
…h-project#7988)

* Adding first draft of delimited-payload token filter WIP

Signed-off-by: Anton Rubin <[email protected]>

* adding the rest of the details to delimited payload token filter

Signed-off-by: Anton Rubin <[email protected]>

* Update delimited-payload.md

Signed-off-by: AntonEliatra <[email protected]>

* updating parameter table

Signed-off-by: Anton Rubin <[email protected]>

* Apply suggestions from code review

Co-authored-by: kolchfa-aws <[email protected]>
Signed-off-by: AntonEliatra <[email protected]>

* Update delimited-payload.md

Signed-off-by: AntonEliatra <[email protected]>

* Apply suggestions from code review

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: AntonEliatra <[email protected]>

* fixing broken link

Signed-off-by: Anton Rubin <[email protected]>

---------

Signed-off-by: Anton Rubin <[email protected]>
Signed-off-by: AntonEliatra <[email protected]>
Co-authored-by: kolchfa-aws <[email protected]>
Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Eric Pugh <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
3 - Tech review PR: Tech review in progress analyzers backport 2.17 Backport for version 2.17 Needs SME Waiting on input from subject matter expert
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Token filters - delimited_payload [DOC]
4 participants