Update DLI docs and docstring

docs/en/aws-deploy-elastic-serverless-forwarder.asciidoc

@@ -229,11 +229,30 @@ For `elasticsearch` the following arguments are supported:
   * `args.password` Password of the elasticsearch instance to connect to. Mandatory when `args.api_key` is not provided. Will take precedence over `args.api_key` if both are defined.
   * `args.api_key`:  API key of elasticsearch endpoint in the format `base64encode(api_key_id:api_key_secret)`. Mandatory when `args.username`  and `args.password` are not provided. Will be ignored if `args.username`/`args.password` are defined.
   * `args.es_datastream_name`: Name of data stream or index where logs should be forwarded to. Lambda supports automatic routing of various {aws} service logs to the corresponding data streams for further processing and storage in the {es} cluster. It supports automatic routing of `aws.cloudtrail`, `aws.cloudwatch_logs`, `aws.elb_logs`, `aws.firewall_logs`, `aws.vpcflow`, and `aws.waf` logs. For other log types, if using data streams, you can optionally set its value in the configuration file according to the naming convention for data streams and available integrations. If the `es_datastream_name` is not specified and it cannot be matched with any of the above {aws} services, then the value will be set to `logs-generic-default`. In versions **v0.29.1** and below, this configuration parameter was named `es_index_or_datastream_name`. Rename the configuration parameter to `es_datastream_name` in your `config.yaml` file on the S3 bucket to continue using it in the future version. The older name `es_index_or_datastream_name` is deprecated as of version **v0.30.0**. The related backward compatibility code is removed from version **v1.0.0**.
-  * `args.es_dead_letter_index`: Name of data stream or index where logs should be redirected to, in case indexing to `args.es_datastream_name` returned an error.
+  * `args.es_dead_letter_index`: Name of data stream or index where logs should be redirected to, in case indexing to `args.es_datastream_name` returned an error. The elasticseach output will NOT forward retryable errors (connection failures, HTTP status code 429) to the dead letter index.  
   * `args.batch_max_actions`: (Optional) Maximum number of actions to send in a single bulk request. Default value: 500.
   * `args.batch_max_bytes`: (Optional) Maximum size in bytes to send in a single bulk request. Default value: 10485760 (10MB).
   * `args.ssl_assert_fingerprint`: (Optional) SSL fingerprint for self-signed SSL certificate on HTTPS transport. The default value is an empty string, meaning the HTTP client requires a valid certificate.
 
+. Here is a sample error indexed in the dead letter index:
++
+[source, json]
+----
+{
+  "@timestamp": "2024-10-07T05:57:59.448925Z",
+  "message": "{\"hey\":{\"message\":\"hey there\"},\"_id\":\"e6542822-4583-438d-9b4d-1a3023b5eeb9\",\"_op_type\":\"create\",\"_index\":\"logs-succeed.pr793-default\"}",
+  "error": {
+    "message": "[1:30] failed to parse field [hey] of type [keyword] in document with id 'e6542822-4583-438d-9b4d-1a3023b5eeb9'. Preview of field's value: '{message=hey there}'",
+    "type": "document_parsing_exception"
+  },
+  "http": {
+    "response": {
+      "status_code": 400
+    }
+  }
+}
+----
+
 For `logstash` the following arguments are supported:
 
   * `args.logstash_url`: URL of {ls} endpoint in the format `http(s)://host:port`

shippers/es.py

@@ -296,21 +296,24 @@ def flush(self) -> None:
 
     def _send_dead_letter_index(self, actions: list[Any]) -> list[Any]:
         """
-        Send the failed actions to the dead letter index (DLI).
+        Index the failed actions in the dead letter index (DLI).
 
-        This function attempts to forward failed actions to the DLI, but may not do so
+        This function attempts to index failed actions to the DLI, but may not do so
         for one of the following reasons:
 
-        1. The action response does not have an HTTP status (e.g., the connection failed).
-        2. The list of action errors to forward is not empty, and the action error type is not in the list.
-        3. The action could not be encoded for indexing in the DLI.
-        4. The action failed indexing attempt in the DLI.
+        1. The failed action could not be encoded for indexing in the DLI.
+        2. ES returned an error on the attempt to index the failed action in the DLI.
+        3. The failed action error is retryable (connection error or status code 429).
+
+        Retryable errors are not indexed in the DLI, as they are expected to be
+        sent again to the data stream at `es_datastream_name` by the replay handler.
 
         Args:
-            actions (list[Any]): A list of actions to be processed.
+            actions (list[Any]): A list of actions to index in the DLI.
 
         Returns:
-            list[Any]: A list of actions that were not indexed in the DLI.
+            list[Any]: A list of actions that were not indexed in the DLI due to one of
+            the reasons mentioned above.
         """
         non_indexed_actions: list[Any] = []
         encoded_actions = []