Add documentation for max_number_processors (#8157)

* Add documentation for max_number_processors Signed-off-by: Rai <[email protected]> * Refined the documentation Signed-off-by: Rai <[email protected]> * Doc review Signed-off-by: Melissa Vagi <[email protected]> * Update _ingest-pipelines/processors/index-processors.md Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Melissa Vagi <[email protected]> * Update _ingest-pipelines/processors/index-processors.md Co-authored-by: Nathan Bower <[email protected]> Signed-off-by: Melissa Vagi <[email protected]> --------- Signed-off-by: Rai <[email protected]> Signed-off-by: Melissa Vagi <[email protected]> Co-authored-by: Rai <[email protected]> Co-authored-by: Melissa Vagi <[email protected]> Co-authored-by: Nathan Bower <[email protected]>
opensearch-project · Sep 11, 2024 · 41f62e0 · 41f62e0
1 parent 9435b46
commit 41f62e0
Showing 1 changed file with 6 additions and 0 deletions.
diff --git a/_ingest-pipelines/processors/index-processors.md b/_ingest-pipelines/processors/index-processors.md
@@ -69,6 +69,12 @@ Processor type | Description
 `urldecode` | Decodes a string from URL-encoded format.
 `user_agent` | Extracts details from the user agent sent by a browser to its web requests. 
 
+## Processor limit settings
+
+You can limit the number of ingest processors using the cluster setting `cluster.ingest.max_number_processors`. The total number of processors includes both the number of processors and the number of [`on_failure`]({{site.url}}{{site.baseurl}}/ingest-pipelines/pipeline-failures/) processors.
+
+The default value for `cluster.ingest.max_number_processors` is `Integer.MAX_VALUE`. Adding a higher number of processors than the value configured in `cluster.ingest.max_number_processors` will throw an `IllegalStateException`.
+
 ## Batch-enabled processors
 
 Some processors support batch ingestion---they can process multiple documents at the same time as a batch. These batch-enabled processors usually provide better performance when using batch processing. For batch processing, use the [Bulk API]({{site.url}}{{site.baseurl}}/api-reference/document-apis/bulk/) and provide a `batch_size` parameter. All batch-enabled processors have a batch mode and a single-document mode. When you ingest documents using the `PUT` method, the processor functions in single-document mode and processes documents in series. Currently, only the `text_embedding` and `sparse_encoding` processors are batch enabled. All other processors process documents one at a time.