Add verbose pipeline parameter to output each processor's execution details

[junweid62]

Jan 16 revision:

1. Add two new fields: status and error

• status: Represents the execution status of the processor, with possible values such as SUCCESS and FAIL.
• error: Captures the error message or details if the processor encounters an issue during execution. If no error occurs, this field remains empty.

2. When verbose pipeline is enabled, errors in a processor will not interrupt the search but will document the error and continue searching

When the verbose pipeline mode is enabled, if a processor encounters an error during execution, the search process will not be interrupted. Instead, the error will be documented in the processor's execution details (e.g., in ProcessorExecutionDetail) and the remaining search process will proceed as normal.

{
            "processor_name": "rename_field",
            "duration_millis": 0,
            "status": "fail",
            "error": "Document with id 1 is missing field messag123e",
            "input_data": [
                {
                    "_index": "my_index",
                    "_id": "1",
                    "_score": 1.0,
                    "_source": {
                        "message": "This is a public message",
                        "visibility": "public"
                    }
                },
                {
                    "_index": "my_index",
                    "_id": "2",
                    "_score": 1.0,
                    "_source": {
                        "message": "This is a private message",
                        "visibility": "private"
                    }
                }
            ],
            "output_data": null
        },
        {

Description

Related RFC : #16705

This PR introduces enhancements to OpenSearch's search pipeline functionality, focusing on improving the traceability and debugging of search request and response transformations. It addresses the increasing complexity of search pipeline processors by implementing verbose mode support, which provides detailed insights into processor execution.

1. Adds Verbose Mode for Search Pipelines:

   • Introduced the verbose_pipeline parameter to search requests, default to false.
   • Tracks each processor’s input, output, execution time, and status (success/failure).
   • Provides detailed logs of the data flow through request and response processors.

2. Improves Pipeline Debugging:

   • Captures step-by-step data transformations applied by each processor.
   • Includes execution metadata (e.g., timestamps and elapsed time) in search responses for better analysis.

3. Supports All Pipeline Configurations:

   • Works seamlessly with:
     • Default pipelines configured at the index level.
     • Pipelines explicitly specified in the search request.
     • Ad-hoc pipelines defined inline.

4. Test Framework Enhancements:

   • Added tests for verbose mode to ensure correct functionality and compatibility.

Example output with request processor: filter_query response processor: rename_field and sort

{
    "took": 29,
    "timed_out": false,
    "_shards": {
        "total": 1,
        "successful": 1,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": {
            "value": 1,
            "relation": "eq"
        },
        "max_score": 0.9116078,
        "hits": [
            {
                "_index": "my_index",
                "_id": "2",
                "_score": 0.9116078,
                "_source": {
                    "notification": "This is a public message 2",
                    "visibility": "public"
                }
            }
        ]
    },
    "processor_result": [
        {
            "processor_name": "filter_query",
            "duration_millis": 0,
            "input_data": {
                "query": {
                    "bool": {
                        "must": [
                            {
                                "match": {
                                    "message": {
                                        "query": "This is a public message 1",
                                        "operator": "OR",
                                        "prefix_length": 0,
                                        "max_expansions": 50,
                                        "fuzzy_transpositions": true,
                                        "lenient": false,
                                        "zero_terms_query": "NONE",
                                        "auto_generate_synonyms_phrase_query": true,
                                        "boost": 1.0
                                    }
                                }
                            }
                        ],
                        "adjust_pure_negative": true,
                        "boost": 1.0
                    }
                },
                "search_pipeline": "my_pipeline"
            },
            "output_data": {
                "query": {
                    "bool": {
                        "must": [
                            {
                                "bool": {
                                    "must": [
                                        {
                                            "match": {
                                                "message": {
                                                    "query": "This is a public message 1",
                                                    "operator": "OR",
                                                    "prefix_length": 0,
                                                    "max_expansions": 50,
                                                    "fuzzy_transpositions": true,
                                                    "lenient": false,
                                                    "zero_terms_query": "NONE",
                                                    "auto_generate_synonyms_phrase_query": true,
                                                    "boost": 1.0
                                                }
                                            }
                                        }
                                    ],
                                    "adjust_pure_negative": true,
                                    "boost": 1.0
                                }
                            }
                        ],
                        "filter": [
                            {
                                "term": {
                                    "visibility": {
                                        "value": "public",
                                        "boost": 1.0
                                    }
                                }
                            }
                        ],
                        "adjust_pure_negative": true,
                        "boost": 1.0
                    }
                },
                "search_pipeline": "my_pipeline"
            }
        },
        {
            "processor_name": "rename_field",
            "duration_millis": 0,
            "input_data": [
                {
                    "_index": "my_index",
                    "_id": "2",
                    "_score": 0.9116078,
                    "_source": {
                        "message": "This is a public message 2",
                        "visibility": "public"
                    }
                }
            ],
            "output_data": [
                {
                    "_index": "my_index",
                    "_id": "2",
                    "_score": 0.9116078,
                    "_source": {
                        "notification": "This is a public message 2",
                        "visibility": "public"
                    }
                }
            ]
        },
        {
            "processor_name": "sort",
            "duration_millis": 0,
            "input_data": [
                {
                    "_index": "my_index",
                    "_id": "2",
                    "_score": 0.9116078,
                    "_source": {
                        "notification": "This is a public message 2",
                        "visibility": "public"
                    }
                }
            ],
            "output_data": [
                {
                    "_index": "my_index",
                    "_id": "2",
                    "_score": 0.9116078,
                    "_source": {
                        "notification": "This is a public message 2",
                        "visibility": "public"
                    }
                }
            ]
        }
    ]
}

Related Issues

Resolves #14745

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

✅ Gradle check result for 021ecec: SUCCESS

[owaiskazi19]

Good start.
Few questions

1. Search pipelines can be used 3 ways: using it as default pipeline, as a temporary search request and through query param. Before applying the verbose pipeline param, we should check if search pipeline is applied in the Search Request. You have mentioned that in the RFC in resolvePipeline but I don't see the check in the PR.

2. Once the check is done then need 3 more tests for verbose pipeline execution with all the ways mentioned above

We are touching few core search files. @msfroh can you take a look at it too?

[owaiskazi19]

This is a general BWC dance we do for such cases, we don't need an issue to track this. @junweid62 can take care of it by following the below

1. Do this on main with onOrAfter(Version.V_3_0_0)). Get it merged.
2. You'll need a manual backport to 2.x, where you do onOrAfter(Version.V_2_19_0). Don't get it merged right away.
3. Before merging the backport to 2.x, open another PR on main to change it to onOrAfter(Version.V_2_19_0).
4. Merge the backport PR.
5. Merge the main version update PR.

[ohltyler]

Tracks each processor’s input, output, execution time, and status (success/failure).

Let's make sure to include the status, plus any associated error message at a per-processor granularity as well.

[github-actions]

❌ Gradle check result for 7276e6c: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[junweid62]

Hi @msfroh, thank you for your detailed feedback. I have addressed all your comments and made the suggested changes, please take a look and let me know if any further adjustments are needed.

[github-actions]

❌ Gradle check result for 2c5759d: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 2c16992: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

✅ Gradle check result for 42e50d0: SUCCESS

[owaiskazi19]

Almost there! Good work @junweid62

[owaiskazi19]

Why this extra check for verbosePipeline?

[junweid62]

Because when verbosePipeline is enabled, we don't interrupt the search process or immediately return on failure. Instead, we log the failure for the processor and continue executing the rest of the pipeline.

[github-actions]

❌ Gradle check result for ba933c2: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[owaiskazi19]

Why are we keeping this as public? Also, not matching with the comment on L35

[junweid62]

This class is intentionally marked as @publicapi because it is part of the search response and will be exposed to users. I understand the confusion with the comment on L35, and I will revise the comment to make it clearer that this class is indeed a public-facing component. :)