From b30d903e6969f10e30e997c768fe88612e94448c Mon Sep 17 00:00:00 2001 From: Rai Date: Mon, 2 Sep 2024 11:57:12 +0530 Subject: [PATCH 1/5] Add documentation for max_number_processors Signed-off-by: Rai --- _ingest-pipelines/processors/index-processors.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/_ingest-pipelines/processors/index-processors.md b/_ingest-pipelines/processors/index-processors.md index 0e1ee1e114..1e8b28fd30 100644 --- a/_ingest-pipelines/processors/index-processors.md +++ b/_ingest-pipelines/processors/index-processors.md @@ -69,6 +69,12 @@ Processor type | Description `urldecode` | Decodes a string from URL-encoded format. `user_agent` | Extracts details from the user agent sent by a browser to its web requests. +## Validations on Processors + +We can configure the limits on the number of ingest processors that should be used. The limit can be configured using the attribute `cluster.ingest.max_number_processors`. The sum of the number of processors and the number of `on_failure` processors are considered for counting the total number of processors on which the limit would be applied. + +The default value for `cluster.ingest.max_number_processors` is `Integer.MAX_VALUE`. If you try to add a number of processors greater than the value configured in `cluster.ingest.max_number_processors`, an `IllegalStateException` will be thrown. + ## Batch-enabled processors Some processors support batch ingestion---they can process multiple documents at the same time as a batch. These batch-enabled processors usually provide better performance when using batch processing. For batch processing, use the [Bulk API]({{site.url}}{{site.baseurl}}/api-reference/document-apis/bulk/) and provide a `batch_size` parameter. All batch-enabled processors have a batch mode and a single-document mode. When you ingest documents using the `PUT` method, the processor functions in single-document mode and processes documents in series. Currently, only the `text_embedding` and `sparse_encoding` processors are batch enabled. All other processors process documents one at a time. From e1c02967bd6805c7b1422c208106ba10443b5d36 Mon Sep 17 00:00:00 2001 From: Rai Date: Tue, 3 Sep 2024 10:42:45 +0530 Subject: [PATCH 2/5] Refined the documentation Signed-off-by: Rai --- _ingest-pipelines/processors/index-processors.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/_ingest-pipelines/processors/index-processors.md b/_ingest-pipelines/processors/index-processors.md index 1e8b28fd30..ce5a985f85 100644 --- a/_ingest-pipelines/processors/index-processors.md +++ b/_ingest-pipelines/processors/index-processors.md @@ -69,9 +69,9 @@ Processor type | Description `urldecode` | Decodes a string from URL-encoded format. `user_agent` | Extracts details from the user agent sent by a browser to its web requests. -## Validations on Processors +## Processor limit settings -We can configure the limits on the number of ingest processors that should be used. The limit can be configured using the attribute `cluster.ingest.max_number_processors`. The sum of the number of processors and the number of `on_failure` processors are considered for counting the total number of processors on which the limit would be applied. +We can configure the limits on the number of ingest processors that should be used. The limit can be configured using the cluster setting `cluster.ingest.max_number_processors`. The sum of the number of processors and the number of `on_failure` processors are considered for counting the total number of processors on which the limit would be applied. The default value for `cluster.ingest.max_number_processors` is `Integer.MAX_VALUE`. If you try to add a number of processors greater than the value configured in `cluster.ingest.max_number_processors`, an `IllegalStateException` will be thrown. From b8b625bbe0547def0fb9a592f2e5469c9b08b265 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Tue, 3 Sep 2024 09:22:24 -0600 Subject: [PATCH 3/5] Doc review Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/index-processors.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/_ingest-pipelines/processors/index-processors.md b/_ingest-pipelines/processors/index-processors.md index ce5a985f85..06322e4592 100644 --- a/_ingest-pipelines/processors/index-processors.md +++ b/_ingest-pipelines/processors/index-processors.md @@ -71,9 +71,9 @@ Processor type | Description ## Processor limit settings -We can configure the limits on the number of ingest processors that should be used. The limit can be configured using the cluster setting `cluster.ingest.max_number_processors`. The sum of the number of processors and the number of `on_failure` processors are considered for counting the total number of processors on which the limit would be applied. +Configure limits on the number of ingest processors using the cluster setting `cluster.ingest.max_number_processors`. The total number of processors includes both the number of processors and the number of [`on_failure`]({{site.url}}{{site.baseurl}}/ingest-pipelines/pipeline-failures/) processors. -The default value for `cluster.ingest.max_number_processors` is `Integer.MAX_VALUE`. If you try to add a number of processors greater than the value configured in `cluster.ingest.max_number_processors`, an `IllegalStateException` will be thrown. +The default value for `cluster.ingest.max_number_processors` is `Integer.MAX_VALUE`. Adding more processors than the configured value in `cluster.ingest.max_number_processors` will throw an `IllegalStateException`. ## Batch-enabled processors From 585d940f329fb8bdf754bcfd54424612d1d5fc50 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Wed, 11 Sep 2024 08:45:57 -0600 Subject: [PATCH 4/5] Update _ingest-pipelines/processors/index-processors.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/index-processors.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_ingest-pipelines/processors/index-processors.md b/_ingest-pipelines/processors/index-processors.md index 06322e4592..d80fcb382c 100644 --- a/_ingest-pipelines/processors/index-processors.md +++ b/_ingest-pipelines/processors/index-processors.md @@ -71,7 +71,7 @@ Processor type | Description ## Processor limit settings -Configure limits on the number of ingest processors using the cluster setting `cluster.ingest.max_number_processors`. The total number of processors includes both the number of processors and the number of [`on_failure`]({{site.url}}{{site.baseurl}}/ingest-pipelines/pipeline-failures/) processors. +You can limit the number of ingest processors using the cluster setting `cluster.ingest.max_number_processors`. The total number of processors includes both the number of processors and the number of [`on_failure`]({{site.url}}{{site.baseurl}}/ingest-pipelines/pipeline-failures/) processors. The default value for `cluster.ingest.max_number_processors` is `Integer.MAX_VALUE`. Adding more processors than the configured value in `cluster.ingest.max_number_processors` will throw an `IllegalStateException`. From 3b5ea5ba4d867fadd545a603b3774f054b20abb2 Mon Sep 17 00:00:00 2001 From: Melissa Vagi Date: Wed, 11 Sep 2024 08:46:15 -0600 Subject: [PATCH 5/5] Update _ingest-pipelines/processors/index-processors.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi --- _ingest-pipelines/processors/index-processors.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_ingest-pipelines/processors/index-processors.md b/_ingest-pipelines/processors/index-processors.md index d80fcb382c..9628a16728 100644 --- a/_ingest-pipelines/processors/index-processors.md +++ b/_ingest-pipelines/processors/index-processors.md @@ -73,7 +73,7 @@ Processor type | Description You can limit the number of ingest processors using the cluster setting `cluster.ingest.max_number_processors`. The total number of processors includes both the number of processors and the number of [`on_failure`]({{site.url}}{{site.baseurl}}/ingest-pipelines/pipeline-failures/) processors. -The default value for `cluster.ingest.max_number_processors` is `Integer.MAX_VALUE`. Adding more processors than the configured value in `cluster.ingest.max_number_processors` will throw an `IllegalStateException`. +The default value for `cluster.ingest.max_number_processors` is `Integer.MAX_VALUE`. Adding a higher number of processors than the value configured in `cluster.ingest.max_number_processors` will throw an `IllegalStateException`. ## Batch-enabled processors