diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 75ede3f363c..60f44451053 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -326,6 +326,7 @@ /packages/stan @elastic/obs-infraobs-integrations /packages/statsd_input @elastic/obs-infraobs-integrations /packages/stormshield @elastic/sec-deployment-and-devices +/packages/sublime_security @elastic/security-service-integrations /packages/suricata @elastic/sec-deployment-and-devices /packages/symantec_edr_cloud @elastic/security-service-integrations /packages/symantec_endpoint @elastic/security-service-integrations diff --git a/packages/sublime_security/_dev/build/build.yml b/packages/sublime_security/_dev/build/build.yml new file mode 100644 index 00000000000..e2b012548e0 --- /dev/null +++ b/packages/sublime_security/_dev/build/build.yml @@ -0,0 +1,3 @@ +dependencies: + ecs: + reference: git@v8.11.0 diff --git a/packages/sublime_security/_dev/build/docs/README.md b/packages/sublime_security/_dev/build/docs/README.md new file mode 100644 index 00000000000..09611cd549a --- /dev/null +++ b/packages/sublime_security/_dev/build/docs/README.md @@ -0,0 +1,120 @@ +# Sublime Security + +Sublime Security is a programmable, AI-powered, cloud email security platform for Microsoft 365 and Google Workspace environments. It is used to block email attacks such as phishing, BEC, malware, threat hunt, and auto-triage user reports. + +The Sublime Security integration collects data for Audit, Email Message(MDM Schema) and Message Event logs using REST API and AWS-S3 or AWS-SQS: + +- REST API mode - Sublime Security integration collects and parses data from the Sublime Security REST APIs. +- AWS S3 polling mode - Sublime Security writes data to S3 and Elastic Agent polls the S3 bucket by listing its contents and reading new files. +- AWS S3 SQS mode - Sublime Security writes data to S3, S3 pushes a new object notification to SQS, Elastic Agent receives the notification from SQS, and then reads the S3 object. Multiple Agents can be used in this mode. + +## Data streams + +The Sublime Security integration collects three types of logs: + +**[Audit](https://docs.sublime.security/reference/listeventsinauditlog)** - Captures detailed records of all significant actions and changes within the platform, including changes to email security policies, user access to email data, and modifications to email configurations, ensuring traceability and compliance for all operations. + +**[Email Message](https://docs.sublime.security/docs/export-message-mdms)** - Represents the flow of individual emails through the platform, including sender and recipient details, spam filtering outcomes, and overall email disposition, helping to secure and analyze email communication. + +**[Message Event](https://docs.sublime.security/reference/getmessage-1)** - Represents document specific actions taken on emails, like spam detection or rule applications, providing detailed insights into how the platform processes and protects email communications. + +## Requirements + +Elastic Agent must be installed. For more information, refer to the link [here](https://www.elastic.co/guide/en/fleet/current/elastic-agent-installation.html). + +### Installing and managing an Elastic Agent: + +You have a few options for installing and managing an Elastic Agent: + +### Install a Fleet-managed Elastic Agent (recommended): + +With this approach, you install Elastic Agent and use Fleet in Kibana to define, configure, and manage your agents in a central location. We recommend using Fleet management because it makes the management and upgrade of your agents considerably easier. + +### Install Elastic Agent in standalone mode (advanced users): + +With this approach, you install Elastic Agent and manually configure the agent locally on the system where it’s installed. You are responsible for managing and upgrading the agents. This approach is reserved for advanced users only. + +### Install Elastic Agent in a containerized environment: + +You can run Elastic Agent inside a container, either with Fleet Server or standalone. Docker images for all versions of Elastic Agent are available from the Elastic Docker registry, and we provide deployment manifests for running on Kubernetes. + +There are some minimum requirements for running Elastic Agent and for more information, refer to the link [here](https://www.elastic.co/guide/en/fleet/current/elastic-agent-installation.html#_minimum_requirements). + +## Setup + +### To collect data from the Sublime Security API: + +#### Step 1: Go to Platform +- Visit the [Sublime Security Platform](https://platform.sublime.security/) and select `API` in Developers section. + +#### Step 2: Generating the API Key +- Retrieve your `API Key`. This key will be used further in the Elastic integration setup to authenticate and access different Sublime Security Logs. +- `Base URL` of Sublime Security is also required for configuring integration. + +**Note**: Users with the `Admin` role are allowed to access `Audit` logs. For more information, refer [here](https://docs.sublime.security/docs/role-based-access-control-rbac). + +### To collect data from AWS S3 Bucket or AWS SQS: + +#### For AWS S3 Bucket, follow the below steps: +- Create an Amazon S3 bucket. Refer to the link [here](https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html). +- User can set the parameter "Bucket List Prefix" according to the requirement. + +#### For AWS SQS, follow the below steps: +1. If data forwarding to an AWS S3 Bucket hasn't been configured, then first set up an AWS S3 Bucket as mentioned in the above documentation. +2. To set up an SQS queue, follow "Step 1: Create an Amazon SQS queue" mentioned in the [Documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/ways-to-add-notification-config-to-bucket.html). + - While creating an SQS Queue, please provide the same bucket ARN that has been generated after creating an AWS S3 Bucket. +3. Set up event notifications for a S3 bucket. Follow this [link](https://docs.aws.amazon.com/AmazonS3/latest/userguide/enable-event-notifications.html). + - Users have to set the prefix parameter the same as the S3 Bucket List Prefix as created earlier. (for example, `exports/sublime_platform_audit_log/` for a audit data stream). + - Select the event type as s3:ObjectCreated:*, select the destination type SQS Queue, and select the queue that has been created in Step 2. + +**Note**: + - Credentials for the above AWS S3 and SQS input types should be configured using the [link](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-aws-s3.html#aws-credentials-config). + - Data collection via AWS S3 Bucket and AWS SQS are mutually exclusive in this case. + - You can configure a global SQS queue for all data streams or a local SQS queue for each data stream. Configuring data stream specific SQS queues will enable better performance and scalability. Data stream specific SQS queues will always override any global queue definitions for that specific data stream. + +### Enabling the integration in Elastic: + +1. In Kibana go to Management > Integrations. +2. In "Search for integrations" search bar, type Sublime Security. +3. Click on the "Sublime Security" integration from the search results. +4. Click on the "Add Sublime Security" button to add the integration. +5. Enable the Integration to collect logs via AWS S3 or API input. +6. Under the AWS S3 input, there are two types of inputs: using AWS S3 Bucket or using SQS. +7. Add all the required integration configuration parameters, including API Key, Interval, Initial Interval and Page Size for API input and Access Key, Secret Key and Session Token for AWS input type to enable data collection. +8. Click on "Save and continue" to save the integration. + +**Note**: +- The Base URL for Sublime Security cloud customers is `https://api.platform.sublimesecurity.com`. Depending on your type of deployment, yours may be different. +- For SSO users, in addition to access key ID and secret access key, the session token is required to configure integration. For IAM users, the session token is optional and not required. + +## Logs reference + +### Audit + +This is the `audit` dataset. + +#### Example + +{{event "audit"}} + +{{fields "audit"}} + +### Email Message + +This is the `email_message` dataset. + +#### Example + +{{event "email_message"}} + +{{fields "email_message"}} + +### Message Event + +This is the `message_event` dataset. + +#### Example + +{{event "message_event"}} + +{{fields "message_event"}} diff --git a/packages/sublime_security/changelog.yml b/packages/sublime_security/changelog.yml new file mode 100644 index 00000000000..6b875ef8f08 --- /dev/null +++ b/packages/sublime_security/changelog.yml @@ -0,0 +1,6 @@ +# newer versions go on top +- version: 0.1.0 + changes: + - description: Initial release. + type: enhancement + link: https://github.com/elastic/integrations/pull/10805 diff --git a/packages/sublime_security/data_stream/audit/_dev/deploy/docker/docker-compose.yml b/packages/sublime_security/data_stream/audit/_dev/deploy/docker/docker-compose.yml new file mode 100644 index 00000000000..979573f79b3 --- /dev/null +++ b/packages/sublime_security/data_stream/audit/_dev/deploy/docker/docker-compose.yml @@ -0,0 +1,15 @@ +version: '2.3' +services: + sublime_security: + image: docker.elastic.co/observability/stream:v0.15.0 + hostname: sublime_security + ports: + - 8090 + volumes: + - ./files:/files:ro + environment: + PORT: '8090' + command: + - http-server + - --addr=:8090 + - --config=/files/config.yml diff --git a/packages/sublime_security/data_stream/audit/_dev/deploy/docker/files/config.yml b/packages/sublime_security/data_stream/audit/_dev/deploy/docker/files/config.yml new file mode 100644 index 00000000000..819414b3ab6 --- /dev/null +++ b/packages/sublime_security/data_stream/audit/_dev/deploy/docker/files/config.yml @@ -0,0 +1,128 @@ +rules: + - path: /v0/audit-log/events + methods: ['GET'] + query_params: + limit: 1 + offset: 0 + request_headers: + Authorization: + - 'Bearer xxxx' + responses: + - status_code: 200 + headers: + Content-Type: + - 'application/json' + body: |- + {{ minify_json ` + { + "events": [ + { + "id": "bd49af79-0cfb-4184-bd18-b0401d69ac61", + "type": "message_group.search", + "created_at": "2024-08-12T06:04:03.714126Z", + "data": { + "request": { + "id": "6ad202de-0def-423d-a0f2-549402e1a9c9", + "path": "/v0/message-groups", + "method": "GET", + "body": "", + "authentication_method": "api_key", + "ip": "1.128.0.0", + "user_agent": "Go-http-client/1.1", + "api_key_name": "demo mode local" + } + }, + "created_by": { + "id": "6e6eca05-4fea-406b-86d4-b40177e25474", + "active": true, + "first_name": "Demo", + "last_name": "User", + "email_address": "demo@example.com", + "phone_number": null, + "created_at": "2024-07-12T05:13:47.879426Z", + "updated_at": "2024-07-12T05:13:47.879426Z", + "role": "admin", + "is_enrolled": true, + "google_oauth_user_id": "d83rb8et4-refe-fe7t4f8efe", + "microsoft_oauth_user_id": "fhe7t4bgf8-freu-ebfur94ref" + } + } + ], + "count": 1, + "total": 2 + } + `}} + - path: /v0/audit-log/events + methods: ['GET'] + query_params: + limit: 1 + offset: 1 + request_headers: + Authorization: + - "Bearer xxxx" + responses: + - status_code: 200 + headers: + Content-Type: + - 'application/json' + body: |- + {{ minify_json ` + { + "events": [ + { + "id": "bd49af79-0cfj-4184-bd18-b0401d69ac61", + "type": "message_group.search", + "created_at": "2024-08-12T06:04:03.714126Z", + "data": { + "request": { + "id": "6ad202de-0def-423d-a0f2-549402e1a9c9", + "path": "/v0/message-groups", + "method": "GET", + "body": "", + "authentication_method": "api_key", + "ip": "175.16.199.0", + "user_agent": "Go-http-client/1.1", + "api_key_name": "demo mode local-2" + } + }, + "created_by": { + "id": "6e6eca05-4fea-406b-86d4-b40177e25474", + "active": true, + "first_name": "User", + "last_name": "Doe", + "email_address": "user@example.com", + "phone_number": null, + "created_at": "2024-07-12T05:13:47.879426Z", + "updated_at": "2024-07-12T05:13:47.879426Z", + "role": "admin", + "is_enrolled": true, + "google_oauth_user_id": "", + "microsoft_oauth_user_id": "" + } + } + ], + "count": 1, + "total": 2 + } + `}} + - path: /v0/audit-log/events + methods: ['GET'] + query_params: + limit: 1 + offset: 2 + request_headers: + Authorization: + - "Bearer xxxx" + responses: + - status_code: 200 + headers: + Content-Type: + - 'application/json' + body: |- + {{ minify_json ` + { + "events": [], + "count": 0, + "total": 2 + } + `}} \ No newline at end of file diff --git a/packages/sublime_security/data_stream/audit/_dev/test/pipeline/test-audit.log b/packages/sublime_security/data_stream/audit/_dev/test/pipeline/test-audit.log new file mode 100644 index 00000000000..309a4b8b364 --- /dev/null +++ b/packages/sublime_security/data_stream/audit/_dev/test/pipeline/test-audit.log @@ -0,0 +1 @@ +{"id":"26704b44-d1b0-4362-8221-579e604f40cb","type":"message_group.search","created_at":"2024-07-30T05:33:47.725649Z","data":{"request":{"id":"ca817b01-cfaa-40ea-ab80-30b6a8e6ef08","path":"/v1/messages/groups/search","method":"GET","query":{},"body":"","api_key_name":"demo mode key","authentication_method":"api_key","ip":"81.2.69.142","user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"}},"created_by":{"id":"6e6eca05-4fea-406b-86d4-b40177e25474","active":true,"first_name":"Bob","last_name":"User","email_address":"bob@example.com","phone_number":null,"created_at":"2024-07-12T05:13:47.879426Z","updated_at":"2024-07-12T05:13:47.879426Z","role":"admin","is_enrolled":true,"google_oauth_user_id":"","microsoft_oauth_user_id":""}} diff --git a/packages/sublime_security/data_stream/audit/_dev/test/pipeline/test-audit.log-expected.json b/packages/sublime_security/data_stream/audit/_dev/test/pipeline/test-audit.log-expected.json new file mode 100644 index 00000000000..413b0446385 --- /dev/null +++ b/packages/sublime_security/data_stream/audit/_dev/test/pipeline/test-audit.log-expected.json @@ -0,0 +1,112 @@ +{ + "expected": [ + { + "@timestamp": "2024-07-30T05:33:47.725Z", + "ecs": { + "version": "8.11.0" + }, + "event": { + "action": "search", + "id": "26704b44-d1b0-4362-8221-579e604f40cb", + "kind": "event", + "original": "{\"id\":\"26704b44-d1b0-4362-8221-579e604f40cb\",\"type\":\"message_group.search\",\"created_at\":\"2024-07-30T05:33:47.725649Z\",\"data\":{\"request\":{\"id\":\"ca817b01-cfaa-40ea-ab80-30b6a8e6ef08\",\"path\":\"/v1/messages/groups/search\",\"method\":\"GET\",\"query\":{},\"body\":\"\",\"api_key_name\":\"demo mode key\",\"authentication_method\":\"api_key\",\"ip\":\"81.2.69.142\",\"user_agent\":\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36\"}},\"created_by\":{\"id\":\"6e6eca05-4fea-406b-86d4-b40177e25474\",\"active\":true,\"first_name\":\"Bob\",\"last_name\":\"User\",\"email_address\":\"bob@example.com\",\"phone_number\":null,\"created_at\":\"2024-07-12T05:13:47.879426Z\",\"updated_at\":\"2024-07-12T05:13:47.879426Z\",\"role\":\"admin\",\"is_enrolled\":true,\"google_oauth_user_id\":\"\",\"microsoft_oauth_user_id\":\"\"}}", + "type": [ + "info" + ] + }, + "http": { + "request": { + "id": "ca817b01-cfaa-40ea-ab80-30b6a8e6ef08", + "method": "GET" + } + }, + "observer": { + "product": "Sublime Security", + "vendor": "Sublime Security" + }, + "related": { + "ip": [ + "81.2.69.142" + ], + "user": [ + "bob@example.com", + "Bob", + "6e6eca05-4fea-406b-86d4-b40177e25474" + ] + }, + "source": { + "geo": { + "city_name": "London", + "continent_name": "Europe", + "country_iso_code": "GB", + "country_name": "United Kingdom", + "location": { + "lat": 51.5142, + "lon": -0.0931 + }, + "region_iso_code": "GB-ENG", + "region_name": "England" + }, + "ip": "81.2.69.142" + }, + "sublime_security": { + "audit": { + "created_at": "2024-07-30T05:33:47.725Z", + "created_by": { + "active": true, + "created_at": "2024-07-12T05:13:47.879Z", + "email_address": "bob@example.com", + "first_name": "Bob", + "id": "6e6eca05-4fea-406b-86d4-b40177e25474", + "is_enrolled": true, + "last_name": "User", + "role": "admin", + "updated_at": "2024-07-12T05:13:47.879Z" + }, + "data": { + "request": { + "api_key_name": "demo mode key", + "authentication_method": "api_key", + "id": "ca817b01-cfaa-40ea-ab80-30b6a8e6ef08", + "ip": "81.2.69.142", + "method": "GET", + "path": "/v1/messages/groups/search", + "user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36" + } + }, + "id": "26704b44-d1b0-4362-8221-579e604f40cb", + "type": "message_group.search" + } + }, + "tags": [ + "preserve_duplicate_custom_fields" + ], + "url": { + "path": "/v1/messages/groups/search" + }, + "user": { + "domain": "example.com", + "email": "bob@example.com", + "full_name": "Bob User", + "id": "6e6eca05-4fea-406b-86d4-b40177e25474", + "name": "bob", + "roles": [ + "admin" + ] + }, + "user_agent": { + "device": { + "name": "Other" + }, + "name": "Chrome", + "original": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36", + "os": { + "full": "Windows 10", + "name": "Windows", + "version": "10" + }, + "version": "126.0.0.0" + } + } + ] +} \ No newline at end of file diff --git a/packages/sublime_security/data_stream/audit/_dev/test/pipeline/test-common-config.yml b/packages/sublime_security/data_stream/audit/_dev/test/pipeline/test-common-config.yml new file mode 100644 index 00000000000..37e8fa225fd --- /dev/null +++ b/packages/sublime_security/data_stream/audit/_dev/test/pipeline/test-common-config.yml @@ -0,0 +1,3 @@ +fields: + tags: + - preserve_duplicate_custom_fields diff --git a/packages/sublime_security/data_stream/audit/_dev/test/system/test-cel-config.yml b/packages/sublime_security/data_stream/audit/_dev/test/system/test-cel-config.yml new file mode 100644 index 00000000000..0203226c2de --- /dev/null +++ b/packages/sublime_security/data_stream/audit/_dev/test/system/test-cel-config.yml @@ -0,0 +1,13 @@ +input: cel +service: sublime_security +vars: + url: http://{{Hostname}}:{{Port}} + api_key: xxxx +data_stream: + vars: + interval: 5m + page_size: 1 + preserve_original_event: true + preserve_duplicate_custom_fields: true +assert: + hit_count: 2 diff --git a/packages/sublime_security/data_stream/audit/agent/stream/aws-s3.yml.hbs b/packages/sublime_security/data_stream/audit/agent/stream/aws-s3.yml.hbs new file mode 100644 index 00000000000..7bf0e86682e --- /dev/null +++ b/packages/sublime_security/data_stream/audit/agent/stream/aws-s3.yml.hbs @@ -0,0 +1,97 @@ +{{#if collect_s3_logs}} + +{{#if bucket_arn}} +bucket_arn: {{bucket_arn}} +{{/if}} +{{#if number_of_workers}} +number_of_workers: {{number_of_workers}} +{{/if}} +{{#if interval}} +bucket_list_interval: {{interval}} +{{/if}} +{{#if bucket_list_prefix}} +bucket_list_prefix: {{bucket_list_prefix}} +{{/if}} + +{{else}} + +{{#if queue_url}} +queue_url: {{queue_url}} +{{/if}} +{{#if region}} +region: {{region}} +{{/if}} +{{#if visibility_timeout}} +visibility_timeout: {{visibility_timeout}} +{{/if}} +{{#if api_timeout}} +api_timeout: {{api_timeout}} +{{/if}} +{{#if max_number_of_messages}} +max_number_of_messages: {{max_number_of_messages}} +{{/if}} +{{#if file_selectors}} +file_selectors: +{{file_selectors}} +{{/if}} + +{{/if}} + +expand_event_list_from_field: events +content_type: application/json +{{#if access_key_id}} +access_key_id: {{access_key_id}} +{{/if}} +{{#if secret_access_key}} +secret_access_key: {{secret_access_key}} +{{/if}} +{{#if session_token}} +session_token: {{session_token}} +{{/if}} +{{#if shared_credential_file}} +shared_credential_file: {{shared_credential_file}} +{{/if}} +{{#if credential_profile_name}} +credential_profile_name: {{credential_profile_name}} +{{/if}} +{{#if role_arn}} +role_arn: {{role_arn}} +{{/if}} +{{#if external_id}} +external_id: {{external_id}} +{{/if}} +{{#if default_region}} +default_region: {{default_region}} +{{/if}} + +{{#if fips_enabled}} +fips_enabled: {{fips_enabled}} +{{/if}} +{{#if proxy_url}} +proxy_url: {{proxy_url}} +{{/if}} +{{#if ssl}} +ssl: {{ssl}} +{{/if}} +tags: +{{#if collect_s3_logs}} + - collect_s3_logs +{{else}} + - collect_sqs_logs +{{/if}} +{{#if preserve_original_event}} + - preserve_original_event +{{/if}} +{{#if preserve_duplicate_custom_fields}} + - preserve_duplicate_custom_fields +{{/if}} +{{#each tags as |tag|}} + - {{tag}} +{{/each}} +{{#contains "forwarded" tags}} +publisher_pipeline.disable_host: true +{{/contains}} +{{#if processors}} +processors: +{{processors}} +{{/if}} diff --git a/packages/sublime_security/data_stream/audit/agent/stream/cel.yml.hbs b/packages/sublime_security/data_stream/audit/agent/stream/cel.yml.hbs new file mode 100644 index 00000000000..09176ba2488 --- /dev/null +++ b/packages/sublime_security/data_stream/audit/agent/stream/cel.yml.hbs @@ -0,0 +1,79 @@ +config_version: 2 +interval: {{interval}} +{{#if enable_request_tracer}} +resource.tracer.filename: "../../logs/cel/http-request-trace-*.ndjson" +resource.tracer.maxbackups: 5 +{{/if}} +{{#if proxy_url}} +resource.proxy_url: {{proxy_url}} +{{/if}} +{{#if ssl}} +resource.ssl: {{ssl}} +{{/if}} +{{#if http_client_timeout}} +resource.timeout: {{http_client_timeout}} +{{/if}} +resource.url: {{url}} +state: + page_size: {{page_size}} + offset: 0 + api_key: {{api_key}} +redact: + fields: + - api_key +program: | + state.with( + request( + "GET", + state.url.trim_right("/") + "/v0/audit-log/events?" + { + "limit": [string(state.page_size)], + "offset": [string(state.offset)] + }.format_query() + ).with({ + "Header": { + "Authorization": ["Bearer " + state.api_key], + }, + }).do_request().as(resp, resp.StatusCode == 200 ? + bytes(resp.Body).decode_json().as(body,{ + "events": body.events.map(e,{ + "message": e.encode_json() + }), + "offset": size(body.events) > 0 ? int(state.offset) + int(state.page_size) : 0, + "want_more": size(body.events) > 0, + }) + : + { + "events": { + "error": { + "code": string(resp.StatusCode), + "id": string(resp.Status), + "message": "GET:"+( + size(resp.Body) != 0 ? + string(resp.Body) + : + string(resp.Status) + ' (' + string(resp.StatusCode) + ')' + ), + }, + }, + "want_more": false, + "offset": 0, + } + ) + ) +tags: +{{#if preserve_original_event}} + - preserve_original_event +{{/if}} +{{#if preserve_duplicate_custom_fields}} + - preserve_duplicate_custom_fields +{{/if}} +{{#each tags as |tag|}} + - {{tag}} +{{/each}} +{{#contains "forwarded" tags}} +publisher_pipeline.disable_host: true +{{/contains}} +{{#if processors}} +processors: +{{processors}} +{{/if}} diff --git a/packages/sublime_security/data_stream/audit/elasticsearch/ingest_pipeline/default.yml b/packages/sublime_security/data_stream/audit/elasticsearch/ingest_pipeline/default.yml new file mode 100644 index 00000000000..1be49745477 --- /dev/null +++ b/packages/sublime_security/data_stream/audit/elasticsearch/ingest_pipeline/default.yml @@ -0,0 +1,414 @@ +--- +description: Pipeline for processing audit logs. +processors: + - set: + field: ecs.version + tag: set_ecs_version + value: 8.11.0 + - fail: + tag: data_collection_error + if: ctx.error?.message != null && ctx.message == null && ctx.event?.original == null + message: error message set and no data to process. + - rename: + field: message + tag: rename_message_to_event_original + target_field: event.original + ignore_missing: true + description: Renames the original `message` field to `event.original` to store a copy of the original message. The `event.original` field is not touched if the document already has one; it may happen when Logstash sends the document. + if: ctx.event?.original == null + - remove: + field: message + tag: remove_message + ignore_missing: true + description: The `message` field is no longer required if the document has an `event.original` field. + if: ctx.event?.original != null + - json: + field: event.original + tag: json_event_original + target_field: json + on_failure: + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - fingerprint: + fields: + - json.created_at + - json.id + tag: fingerprint_audit_event + target_field: _id + ignore_missing: true + - set: + field: event.kind + tag: set_event_kind_to_event + value: event + - append: + field: event.type + tag: append_info_into_event_type + value: info + allow_duplicates: false + - set: + field: observer.vendor + tag: set_observer_vendor + value: Sublime Security + - set: + field: observer.product + tag: set_observer_product + value: Sublime Security + - date: + field: json.created_at + tag: date_created_at + target_field: sublime_security.audit.created_at + formats: + - ISO8601 + if: ctx.json?.created_at != null && ctx.json.created_at != '' + on_failure: + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - set: + field: '@timestamp' + tag: set_@timestamp_from_audit_created_at + copy_from: sublime_security.audit.created_at + ignore_empty_value: true + - convert: + field: json.created_by.active + tag: convert_created_by_active_to_boolean + target_field: sublime_security.audit.created_by.active + type: boolean + ignore_missing: true + on_failure: + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - date: + field: json.created_by.created_at + tag: date_created_by_created_at + target_field: sublime_security.audit.created_by.created_at + formats: + - ISO8601 + if: ctx.json?.created_by?.created_at != null && ctx.json.created_by.created_at != '' + on_failure: + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - date: + field: json.created_by.deleted_at + tag: date_created_by_deleted_at + target_field: sublime_security.audit.created_by.deleted_at + formats: + - ISO8601 + if: ctx.json?.created_by?.deleted_at != null && ctx.json.created_by.deleted_at != '' + on_failure: + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - rename: + field: json.created_by.email_address + tag: rename_created_by_email_address + target_field: sublime_security.audit.created_by.email_address + ignore_missing: true + - dissect: + field: sublime_security.audit.created_by.email_address + tag: dissect_created_by_email_address + description: Extract username and domain from email address. + pattern: '%{user.name}@%{user.domain}' + if: ctx.sublime_security?.audit?.created_by?.email_address != null + on_failure: + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag fail-{{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - set: + field: user.email + tag: set_user_email_from_audit_created_by_email_address + copy_from: sublime_security.audit.created_by.email_address + ignore_empty_value: true + - append: + field: related.user + tag: append_audit_created_by_email_address_into_related_user + value: '{{{sublime_security.audit.created_by.email_address}}}' + allow_duplicates: false + if: ctx.sublime_security?.audit?.created_by?.email_address != null + - rename: + field: json.created_by.first_name + tag: rename_created_by_first_name + target_field: sublime_security.audit.created_by.first_name + ignore_missing: true + - append: + field: related.user + tag: append_audit_created_by_first_name_into_related_user + value: '{{{sublime_security.audit.created_by.first_name}}}' + allow_duplicates: false + if: ctx.sublime_security?.audit?.created_by?.email_address != null + - rename: + field: json.created_by.last_name + tag: rename_created_by_last_name + target_field: sublime_security.audit.created_by.last_name + ignore_missing: true + - set: + field: user.full_name + tag: set_user_name_from_audit_created_by_first_name_and_last_name + value: '{{{sublime_security.audit.created_by.first_name}}} {{{sublime_security.audit.created_by.last_name}}}' + if: ctx.sublime_security?.audit?.created_by?.first_name != null && ctx.sublime_security.audit.created_by?.last_name != null + ignore_empty_value: true + - rename: + field: json.created_by.google_oauth_user_id + tag: rename_created_by_google_oauth_user_id + target_field: sublime_security.audit.created_by.google_oauth_user_id + ignore_missing: true + - append: + field: related.user + tag: append_audit_created_by_google_oauth_user_id_into_related_user + value: '{{{sublime_security.audit.created_by.google_oauth_user_id}}}' + allow_duplicates: false + if: ctx.sublime_security?.audit?.created_by?.google_oauth_user_id != null + - rename: + field: json.created_by.id + tag: rename_created_by_id + target_field: sublime_security.audit.created_by.id + ignore_missing: true + - set: + field: user.id + tag: set_user_id_from_audit_created_by_id + copy_from: sublime_security.audit.created_by.id + ignore_empty_value: true + - append: + field: related.user + tag: append_audit_created_by_id_into_related_user + value: '{{{sublime_security.audit.created_by.id}}}' + allow_duplicates: false + if: ctx.sublime_security?.audit?.created_by?.id != null + - convert: + field: json.created_by.is_enrolled + tag: convert_created_by_is_enrolled_to_boolean + target_field: sublime_security.audit.created_by.is_enrolled + type: boolean + ignore_missing: true + on_failure: + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - rename: + field: json.created_by.microsoft_oauth_user_id + tag: rename_created_by_microsoft_oauth_user_id + target_field: sublime_security.audit.created_by.microsoft_oauth_user_id + ignore_missing: true + - append: + field: related.user + tag: append_audit_created_by_microsoft_oauth_user_id_into_related_user + value: '{{{sublime_security.audit.created_by.microsoft_oauth_user_id}}}' + allow_duplicates: false + if: ctx.sublime_security?.audit?.created_by?.microsoft_oauth_user_id != null + - rename: + field: json.created_by.phone_number + tag: rename_created_by_phone_number + target_field: sublime_security.audit.created_by.phone_number + ignore_missing: true + - rename: + field: json.created_by.role + tag: rename_created_by_role + target_field: sublime_security.audit.created_by.role + ignore_missing: true + - append: + field: user.roles + tag: append_sublime_security_audit_created_by_role_into_user_roles + value: '{{{sublime_security.audit.created_by.role}}}' + allow_duplicates: false + if: ctx.sublime_security?.audit?.created_by?.role != null + - date: + field: json.created_by.updated_at + tag: date_created_by_updated_at + target_field: sublime_security.audit.created_by.updated_at + formats: + - ISO8601 + if: ctx.json?.created_by?.updated_at != null && ctx.json.created_by.updated_at != '' + on_failure: + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - rename: + field: json.data.message.id + tag: rename_data_message_id + target_field: sublime_security.audit.data.message.id + ignore_missing: true + - rename: + field: json.data.message_group.id + tag: rename_data_message_group_id + target_field: sublime_security.audit.data.message_group.id + ignore_missing: true + - rename: + field: json.data.request.api_key_name + tag: rename_data_request_api_key_name + target_field: sublime_security.audit.data.request.api_key_name + ignore_missing: true + - rename: + field: json.data.request.authentication_method + tag: rename_data_request_authentication_method + target_field: sublime_security.audit.data.request.authentication_method + ignore_missing: true + - rename: + field: json.data.request.body + tag: rename_data_request_body + target_field: sublime_security.audit.data.request.body + ignore_missing: true + - set: + field: http.request.body.content + tag: set_http_request_body_content_from_audit_data_request_body + copy_from: sublime_security.audit.data.request.body + ignore_empty_value: true + - rename: + field: json.data.request.id + tag: rename_data_request_id + target_field: sublime_security.audit.data.request.id + ignore_missing: true + - set: + field: http.request.id + tag: set_http_request_id_from_audit_data_request_id + copy_from: sublime_security.audit.data.request.id + ignore_empty_value: true + - convert: + field: json.data.request.ip + tag: convert_data_request_ip_to_ip + target_field: sublime_security.audit.data.request.ip + type: ip + ignore_missing: true + if: ctx.json?.data?.request?.ip != null && ctx.json.data.request.ip != '' + on_failure: + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - set: + field: source.ip + tag: set_source_ip_from_audit_data_request_ip + copy_from: sublime_security.audit.data.request.ip + ignore_empty_value: true + - geoip: + field: source.ip + target_field: source.geo + tag: geoip_source_ip + ignore_missing: true + - append: + field: related.ip + tag: append_source_ip_into_related_ip + value: '{{{source.ip}}}' + allow_duplicates: false + if: ctx.source?.ip != null + - rename: + field: json.data.request.method + tag: rename_data_request_method + target_field: sublime_security.audit.data.request.method + ignore_missing: true + - set: + field: http.request.method + tag: set_http_request_method_from_audit_data_request_method + copy_from: sublime_security.audit.data.request.method + ignore_empty_value: true + - rename: + field: json.data.request.path + tag: rename_data_request_path + target_field: sublime_security.audit.data.request.path + ignore_missing: true + - set: + field: url.path + tag: set_url_path_from_audit_data_request_path + copy_from: sublime_security.audit.data.request.path + ignore_empty_value: true + - rename: + field: json.data.request.query + tag: rename_data_request_query + target_field: sublime_security.audit.data.request.query + ignore_missing: true + - script: + lang: painless + description: Painless script to set url_query. + tag: painless_set_url_query + if: ctx.sublime_security?.audit?.data?.request?.query != null + source: |- + StringBuilder sb = new StringBuilder(); + Map attributes = ctx.sublime_security.audit.data.request.query; + for (entry in attributes.entrySet()) { + sb.append(entry.getKey()).append('=').append(entry.getValue()).append('&'); + } + if(ctx.url == null) { + ctx.put("url", new HashMap()); + } + ctx.url.query = sb.length() > 0 ? sb.substring(0, sb.length() - 1) : null; + - rename: + field: json.data.request.user_agent + tag: rename_data_request_user_agent + target_field: sublime_security.audit.data.request.user_agent + ignore_missing: true + - user_agent: + field: sublime_security.audit.data.request.user_agent + ignore_missing: true + - rename: + field: json.id + tag: rename_id + target_field: sublime_security.audit.id + ignore_missing: true + - set: + field: event.id + tag: set_event_id_from_audit_id + copy_from: sublime_security.audit.id + ignore_empty_value: true + - rename: + field: json.type + tag: rename_type + target_field: sublime_security.audit.type + ignore_missing: true + - grok: + field: sublime_security.audit.type + tag: grok_event_action_from_audit_type + patterns: + - '.*.%{WORD:event.action}' + ignore_missing: true + - remove: + field: + - sublime_security.audit.created_at + - sublime_security.audit.created_by.email_address + - sublime_security.audit.created_by.id + - sublime_security.audit.created_by.role + - sublime_security.audit.data.request.body + - sublime_security.audit.data.request.id + - sublime_security.audit.data.request.ip + - sublime_security.audit.data.request.method + - sublime_security.audit.data.request.path + - sublime_security.audit.data.request.user_agent + - sublime_security.audit.id + tag: remove_custom_duplicate_fields + ignore_missing: true + if: ctx.tags == null || !(ctx.tags.contains('preserve_duplicate_custom_fields')) + - remove: + field: json + tag: remove_json + ignore_missing: true + - script: + lang: painless + description: Drops null/empty values recursively. + tag: painless_remove_null + source: |- + boolean drop(Object object) { + if (object == null || object == '') { + return true; + } else if (object instanceof Map) { + ((Map) object).values().removeIf(v -> drop(v)); + return (((Map) object).size() == 0); + } else if (object instanceof List) { + ((List) object).removeIf(v -> drop(v)); + return (((List) object).length == 0); + } + return false; + } + drop(ctx); + - set: + field: event.kind + value: pipeline_error + tag: set_pipeline_error_into_event_kind + if: ctx.error?.message != null +on_failure: + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - set: + field: event.kind + tag: set_pipeline_error_to_event_kind + value: pipeline_error diff --git a/packages/sublime_security/data_stream/audit/fields/base-fields.yml b/packages/sublime_security/data_stream/audit/fields/base-fields.yml new file mode 100644 index 00000000000..f9c75ce55dd --- /dev/null +++ b/packages/sublime_security/data_stream/audit/fields/base-fields.yml @@ -0,0 +1,20 @@ +- name: data_stream.type + type: constant_keyword + description: Data stream type. +- name: data_stream.dataset + type: constant_keyword + description: Data stream dataset. +- name: data_stream.namespace + type: constant_keyword + description: Data stream namespace. +- name: event.module + type: constant_keyword + description: Event module. + value: sublime_security +- name: event.dataset + type: constant_keyword + description: Event dataset. + value: sublime_security.audit +- name: '@timestamp' + type: date + description: Event timestamp. diff --git a/packages/sublime_security/data_stream/audit/fields/beats.yml b/packages/sublime_security/data_stream/audit/fields/beats.yml new file mode 100644 index 00000000000..4084f1dc7f5 --- /dev/null +++ b/packages/sublime_security/data_stream/audit/fields/beats.yml @@ -0,0 +1,6 @@ +- name: input.type + type: keyword + description: Type of filebeat input. +- name: log.offset + type: long + description: Log offset. diff --git a/packages/sublime_security/data_stream/audit/fields/fields.yml b/packages/sublime_security/data_stream/audit/fields/fields.yml new file mode 100644 index 00000000000..6cef6ed7808 --- /dev/null +++ b/packages/sublime_security/data_stream/audit/fields/fields.yml @@ -0,0 +1,102 @@ +- name: sublime_security + type: group + fields: + - name: audit + type: group + fields: + - name: created_at + type: date + description: Event creation time. + - name: created_by + type: group + fields: + - name: active + type: boolean + - name: created_at + type: date + description: User creation time. + - name: deleted_at + type: date + description: User deletion time. + - name: email_address + type: keyword + description: Email address. + - name: first_name + type: keyword + description: First name. + - name: google_oauth_user_id + type: keyword + description: The user's Google user ID, if it exists. + - name: id + type: keyword + description: User ID. + - name: is_enrolled + type: boolean + description: Whether the user has begun using the system (e.g. accepted an invitation or logged in at least once). + - name: last_name + type: keyword + description: Last name. + - name: microsoft_oauth_user_id + type: keyword + description: The user's Microsoft user ID, if it exists. + - name: phone_number + type: keyword + description: Phone number. + - name: role + type: keyword + description: Role assumed by the user. + - name: updated_at + type: date + description: User last updated time. + - name: data + type: group + fields: + - name: message + type: group + fields: + - name: id + type: keyword + description: Message ID. + - name: message_group + type: group + fields: + - name: id + type: keyword + description: Message Group ID. + - name: request + type: group + fields: + - name: api_key_name + type: keyword + description: Name of API key if an API key was used. + - name: authentication_method + type: keyword + description: Description of how request was authenticated. + - name: body + type: keyword + description: Request body. + - name: id + type: keyword + description: API request ID. + - name: ip + type: ip + description: IP address of requester, if available. + - name: method + type: keyword + description: HTTP method. + - name: path + type: keyword + description: URL path. + - name: query + type: object + object_type: keyword + description: Query parameters. + - name: user_agent + type: keyword + description: User agent of requester, if available. + - name: id + type: keyword + description: Event ID. + - name: type + type: keyword + description: Event type. diff --git a/packages/sublime_security/data_stream/audit/manifest.yml b/packages/sublime_security/data_stream/audit/manifest.yml new file mode 100644 index 00000000000..6d2fd8a8c0e --- /dev/null +++ b/packages/sublime_security/data_stream/audit/manifest.yml @@ -0,0 +1,188 @@ +title: Sublime Security Audit logs +type: logs +streams: + - input: cel + title: Sublime Security Audit logs + description: Collecting Sublime Security Audit logs via API. + enabled: false + template_path: cel.yml.hbs + vars: + - name: interval + type: text + title: Interval + description: Duration between requests to the Sublime Security API. Supported units for this parameter are h/m/s. + default: 1h + multi: false + required: true + show_user: true + - name: page_size + type: text + title: Page Size + multi: false + required: true + show_user: false + description: 'Page size for the response of the Sublime Security API. Note: The maximum limit is 500.' + default: 500 + - name: http_client_timeout + type: text + title: HTTP Client Timeout + description: Duration before declaring that the HTTP client connection has timed out. Supported time units are ns, us, ms, s, m, h. + multi: false + required: true + show_user: false + default: 30s + - name: enable_request_tracer + type: bool + title: Enable request tracing + multi: false + required: false + show_user: false + description: >- + The request tracer logs requests and responses to the agent's local file-system for debugging configurations. Enabling this request tracing compromises security and should only be used for debugging. See [documentation](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-cel.html#_resource_tracer_filename) for details. + - name: tags + type: text + title: Tags + multi: true + required: true + show_user: false + default: + - forwarded + - sublime_security-audit + - name: preserve_original_event + required: false + show_user: true + title: Preserve original event + description: Preserves a raw copy of the original event, added to the field `event.original`. + type: bool + multi: false + default: false + - name: preserve_duplicate_custom_fields + required: false + show_user: false + title: Preserve duplicate custom fields + description: Preserve sublime_security.audit fields that were copied to Elastic Common Schema (ECS) fields. + type: bool + multi: false + default: false + - name: processors + type: yaml + title: Processors + multi: false + required: false + show_user: false + description: >- + Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. + - input: aws-s3 + template_path: aws-s3.yml.hbs + title: Sublime Security Audit logs via AWS S3 or SQS + description: Collecting Sublime Security Audit logs via AWS S3 or SQS input. + enabled: false + vars: + - name: bucket_arn + type: text + title: '[S3] Bucket ARN' + multi: false + required: false + show_user: true + description: ARN of the AWS S3 bucket that will be polled for list operation. It is a required parameter for collecting logs via the AWS S3. + - name: bucket_list_prefix + type: text + title: '[S3] Bucket Prefix' + multi: false + required: false + show_user: true + description: Prefix to apply for the list request to the S3 bucket. + - name: interval + type: text + title: '[S3] Interval' + multi: false + required: false + show_user: true + default: 120s + description: Listing of the S3 bucket will be polled according to the time interval defined by bucket_list_interval config. Default value is 120 secs. Supported units for this parameter are h/m/s. + - name: number_of_workers + type: integer + title: '[S3] Number of Workers' + multi: false + required: false + show_user: true + default: 5 + description: Number of workers that will process the S3 objects listed. + - name: queue_url + type: text + title: '[SQS] Queue URL' + multi: false + required: false + show_user: true + description: URL of the AWS SQS queue that messages will be received from. It is a required parameter for collecting logs via the AWS SQS. + - name: visibility_timeout + type: text + title: '[SQS] Visibility Timeout' + multi: false + required: false + show_user: true + default: 300s + description: The duration that the received messages are hidden from subsequent retrieve requests after being retrieved by a ReceiveMessage request. The maximum is 12 hours. Supported units for this parameter are h/m/s. + - name: api_timeout + type: text + title: '[SQS] API Timeout' + multi: false + required: false + show_user: true + default: 120s + description: The maximum duration of AWS API can take. The maximum is half of the visibility timeout value. Supported units for this parameter are h/m/s. + - name: max_number_of_messages + type: integer + title: '[SQS] Maximum Concurrent SQS Messages' + required: false + show_user: true + default: 5 + description: The maximum number of SQS messages that can be inflight at any time. + - name: file_selectors + type: yaml + title: '[SQS] File Selectors' + multi: false + required: false + show_user: false + description: >- + If the SQS queue will have events that correspond to files that this integration shouldn’t process, file_selectors can be used to limit the files that are downloaded. This is a list of selectors which are made up of regex and expand_event_list_from_field options. The regex should match the S3 object key in the SQS message, and the optional expand_event_list_from_field is the same as the global setting. If file_selectors is given, then any global expand_event_list_from_field value is ignored in favor of the ones specified in the file_selectors. Regexes use [RE2 syntax](https://pkg.go.dev/regexp/syntax). Files that do not match one of the regexes will not be processed. + - name: external_id + type: text + title: External ID + multi: false + required: false + show_user: false + description: External ID to use when assuming a role in another account. + - name: tags + type: text + title: Tags + multi: true + required: true + show_user: false + default: + - forwarded + - sublime_security-audit + - name: preserve_original_event + required: false + show_user: true + title: Preserve original event + description: Preserves a raw copy of the original event, added to the field `event.original`. + type: bool + multi: false + default: false + - name: preserve_duplicate_custom_fields + required: true + show_user: false + title: Preserve duplicate custom fields + description: Preserve sublime_security.audit fields that were mapped to Elastic Common Schema (ECS) fields. + type: bool + multi: false + default: false + - name: processors + type: yaml + title: Processors + multi: false + required: false + show_user: false + description: >- + Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See [Processors](https://www.elastic.co/guide/en/beats/filebeat/current/filtering-and-enhancing-data.html) for details. diff --git a/packages/sublime_security/data_stream/audit/sample_event.json b/packages/sublime_security/data_stream/audit/sample_event.json new file mode 100644 index 00000000000..a45b1a79bd2 --- /dev/null +++ b/packages/sublime_security/data_stream/audit/sample_event.json @@ -0,0 +1,121 @@ +{ + "@timestamp": "2024-08-12T06:04:03.714Z", + "agent": { + "ephemeral_id": "390c3f2d-c9eb-4229-9992-0f4fc2436f51", + "id": "5f3fcbb9-1a97-4ff3-857f-167af6664464", + "name": "docker-fleet-agent", + "type": "filebeat", + "version": "8.13.0" + }, + "data_stream": { + "dataset": "sublime_security.audit", + "namespace": "99243", + "type": "logs" + }, + "ecs": { + "version": "8.11.0" + }, + "elastic_agent": { + "id": "5f3fcbb9-1a97-4ff3-857f-167af6664464", + "snapshot": false, + "version": "8.13.0" + }, + "event": { + "action": "search", + "agent_id_status": "verified", + "dataset": "sublime_security.audit", + "id": "bd49af79-0cfb-4184-bd18-b0401d69ac61", + "ingested": "2024-08-28T10:35:52Z", + "kind": "event", + "original": "{\"created_at\":\"2024-08-12T06:04:03.714126Z\",\"created_by\":{\"active\":true,\"created_at\":\"2024-07-12T05:13:47.879426Z\",\"email_address\":\"demo@example.com\",\"first_name\":\"Demo\",\"google_oauth_user_id\":\"d83rb8et4-refe-fe7t4f8efe\",\"id\":\"6e6eca05-4fea-406b-86d4-b40177e25474\",\"is_enrolled\":true,\"last_name\":\"User\",\"microsoft_oauth_user_id\":\"fhe7t4bgf8-freu-ebfur94ref\",\"phone_number\":null,\"role\":\"admin\",\"updated_at\":\"2024-07-12T05:13:47.879426Z\"},\"data\":{\"request\":{\"api_key_name\":\"demo mode local\",\"authentication_method\":\"api_key\",\"body\":\"\",\"id\":\"6ad202de-0def-423d-a0f2-549402e1a9c9\",\"ip\":\"1.128.0.0\",\"method\":\"GET\",\"path\":\"/v0/message-groups\",\"user_agent\":\"Go-http-client/1.1\"}},\"id\":\"bd49af79-0cfb-4184-bd18-b0401d69ac61\",\"type\":\"message_group.search\"}", + "type": [ + "info" + ] + }, + "http": { + "request": { + "id": "6ad202de-0def-423d-a0f2-549402e1a9c9", + "method": "GET" + } + }, + "input": { + "type": "cel" + }, + "observer": { + "product": "Sublime Security", + "vendor": "Sublime Security" + }, + "related": { + "ip": [ + "1.128.0.0" + ], + "user": [ + "demo@example.com", + "Demo", + "d83rb8et4-refe-fe7t4f8efe", + "6e6eca05-4fea-406b-86d4-b40177e25474", + "fhe7t4bgf8-freu-ebfur94ref" + ] + }, + "source": { + "ip": "1.128.0.0" + }, + "sublime_security": { + "audit": { + "created_at": "2024-08-12T06:04:03.714Z", + "created_by": { + "active": true, + "created_at": "2024-07-12T05:13:47.879Z", + "email_address": "demo@example.com", + "first_name": "Demo", + "google_oauth_user_id": "d83rb8et4-refe-fe7t4f8efe", + "id": "6e6eca05-4fea-406b-86d4-b40177e25474", + "is_enrolled": true, + "last_name": "User", + "microsoft_oauth_user_id": "fhe7t4bgf8-freu-ebfur94ref", + "role": "admin", + "updated_at": "2024-07-12T05:13:47.879Z" + }, + "data": { + "request": { + "api_key_name": "demo mode local", + "authentication_method": "api_key", + "id": "6ad202de-0def-423d-a0f2-549402e1a9c9", + "ip": "1.128.0.0", + "method": "GET", + "path": "/v0/message-groups", + "user_agent": "Go-http-client/1.1" + } + }, + "id": "bd49af79-0cfb-4184-bd18-b0401d69ac61", + "type": "message_group.search" + } + }, + "tags": [ + "preserve_original_event", + "preserve_duplicate_custom_fields", + "forwarded", + "sublime_security-audit" + ], + "url": { + "path": "/v0/message-groups" + }, + "user": { + "domain": "example.com", + "email": "demo@example.com", + "full_name": "Demo User", + "id": "6e6eca05-4fea-406b-86d4-b40177e25474", + "name": "demo", + "roles": [ + "admin" + ] + }, + "user_agent": { + "device": { + "name": "Other" + }, + "name": "Go-http-client", + "original": "Go-http-client/1.1", + "version": "1.1" + } +} \ No newline at end of file diff --git a/packages/sublime_security/data_stream/email_message/_dev/deploy/tf/env.yml b/packages/sublime_security/data_stream/email_message/_dev/deploy/tf/env.yml new file mode 100644 index 00000000000..aee5f1c5900 --- /dev/null +++ b/packages/sublime_security/data_stream/email_message/_dev/deploy/tf/env.yml @@ -0,0 +1,9 @@ +version: '2.3' +services: + terraform: + environment: + - AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} + - AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} + - AWS_SESSION_TOKEN=${AWS_SESSION_TOKEN} + - AWS_DEFAULT_PROFILE=${AWS_DEFAULT_PROFILE} + - AWS_REGION=${AWS_REGION:-us-east-1} diff --git a/packages/sublime_security/data_stream/email_message/_dev/deploy/tf/files/test-email-message.log b/packages/sublime_security/data_stream/email_message/_dev/deploy/tf/files/test-email-message.log new file mode 100644 index 00000000000..4019e8bf48a --- /dev/null +++ b/packages/sublime_security/data_stream/email_message/_dev/deploy/tf/files/test-email-message.log @@ -0,0 +1 @@ +{"body":{"plain":{"raw":"Sublime Security test message.\n","charset":"utf-8","content_transfer_encoding":"base64"},"current_thread":{"text":"Sublime Security test message."},"html":{"charset":"utf-8","content_transfer_encoding":"base64","display_text":"Sublime Security test message.","raw":"

Sublime Security test message.

","inner_text":"

Sublime Security test message.

"},"ips":[{"ip":"1.128.0.0"}],"links":[{"display_text":"Click here!","mismatched":true,"display_url":{"fragment":"search","password":"pass123","path":"/test","port":80,"query_params":"q=elasticsearch","rewrite":{"encoders":["base64"],"original":"demo"},"scheme":"https","url":"https://example.com/test?q=elasticsearch#search","username":"test","domain":{"domain":"example.com","punycode":"demo","root_domain":"example.com","subdomain":"example","tld":"com","valid":true,"sld":"example"}}},{"href_url":{"fragment":"search","password":"pass123","path":"/test","port":80,"query_params":"q=elasticsearch","rewrite":{"encoders":["base64"],"original":"demo"},"scheme":"https","url":"https://example.com/test?q=elasticsearch#search","username":"test","domain":{"domain":"example.com","punycode":"demo","root_domain":"example.com","subdomain":"example","tld":"com","valid":true,"sld":"example"}}}]},"external":{"created_at":"2024-08-02T07:40:25.135939305Z","message_id":"2fe271830bbad5fe3a70abbe7a8c0bfe7refe3ffe","route_type":"sent","spam":false,"spam_folder":true,"thread_id":"sample_data"},"attachments":[{"content_id":"abc123","content_transfer_encoding":"base64","content_type":"application/pdf","file_extension":".pdf","file_name":"sample_document.pdf","file_type":"document","md5":"1a2b3c","raw":"JVBERi0xLjMKJcfs4AAQSkZjRgABAQE","sha1":"4d5e6f","sha256":"7g8h9i","size":102400},{"content_id":"xyz456","content_transfer_encoding":"7bit","content_type":"image/jpeg","file_extension":".jpg","file_name":"image_photo.jpg","file_type":"image","md5":"7h8i9j","raw":"/9j/4AAQSkZJRgABAQEJVBERi0xLjMKJd","sha1":"1k2l3m","sha256":"4n5o6p","size":204800},{"content_id":"efg789","content_transfer_encoding":"quoted-printable","content_type":"text/plain","file_extension":".txt","file_name":"notes.txt","file_type":"text","md5":"1x2y3z","raw":"SGVsbG8gdVsbG8gd29yb29ybGQhVsbG8gd29yb","sha1":"4a5b6c","sha256":"7d8e9f","size":5120}],"headers":{"x_authenticated_domain":{"domain":"example.com","punycode":"xn--example-d4a.com","root_domain":"example.com","sld":"example","subdomain":"sub","tld":"com","valid":true},"x_authenticated_sender":{"domain":{"domain":"example.com","punycode":"example.com","root_domain":"example.com","sld":"example","subdomain":"sub","tld":"com","valid":true},"email":"user@example.com","local_part":"user"},"x_client_ip":{"ip":"1.128.0.0"},"x_originating_ip":{"ip":"1.128.0.0"},"x_secure_server_account":"account_value","x_sender":{"domain":{"domain":"example.com","punycode":"example.com","root_domain":"example.com","sld":"example","subdomain":"sub","tld":"com","valid":true},"email":"user@example.com","local_part":"user"},"return_path":{"domain":{"domain":"example.com","punycode":"xn--example-d4a.com","root_domain":"example","sld":"example","subdomain":"sub","tld":"com","valid":true},"email":"user@example.com","local_part":"user"},"references":["test1","test2"],"auth_summary":{"dmarc":{"details":{"action":"quarantine","disposition":"quarantine","from":{"domain":"example.com","punycode":"example.com","root_domain":"example.com","sld":"example","subdomain":"example","tld":"com","valid":true},"policy":"reject","sub_policy":"none","verdict":"pass","version":"1.0"},"pass":true,"received_hop":1},"spf":{"details":{"client_ip":{"ip":"1.128.0.0"},"description":"SPF record found","designator":"pass","helo":{"domain":"example.com","punycode":"example.com","root_domain":"example.com","sld":"example","subdomain":"example","tld":"com","valid":true},"server":{"domain":"mail.example.com","punycode":"mail.example.com","root_domain":"example.com","sld":"example","subdomain":"mail","tld":"com","valid":true},"verdict":"pass"},"error":"true","pass":true,"received_hop":2}},"date":"2019-10-21T18:23:24Z","date_original_offset":"-4","hops":[{"index":0,"fields":[{"name":"To","value":"user@example.com","position":0},{"name":"Subject","value":"Sublime-Security-Standard-Test-String","position":1},{"name":"Date","value":"Mon, 21 Oct 2019 14:23:24 -0400","position":2},{"name":"From","value":"Sublime Security Test ","position":3}],"authentication_results":{"compauth":{"verdict":"pass","reason":"reason_value"},"dkim":"pass","dkim_details":{"algorithm":"rsa-sha256","body_hash":"abcdefg","domain":"example.com","headers":"from, to, subject","instance":"example.com","selector":"abcdefg","signature":"abcdefg","type":"dkim","version":"1.0"},"dmarc":"pass","dmarc_details":{"action":"quarantine","disposition":"quarantine","from":{"domain":"example.com","punycode":"example.com","root_domain":"example.com","sld":"example","subdomain":"example","tld":"com","valid":true},"policy":"reject","sub_policy":"none","verdict":"pass","version":"1.0"},"instance":"example.com","server":{"domain":"mail.example.com","punycode":"mail.example.com","root_domain":"example.com","sld":"example","subdomain":"mail","tld":"com","valid":true},"spf":"pass","spf_details":{"client_ip":{"ip":"1.128.0.0"},"description":"SPF record found","designator":"pass","helo":{"domain":"example.com","punycode":"example.com","root_domain":"example.com","sld":"example","subdomain":"example","tld":"com","valid":true},"server":{"domain":"mail.example.com","root_domain":"example.com","sld":"example","subdomain":"mail","tld":"com","valid":true},"verdict":"pass"},"type":"spf"},"received":{"additional":{"raw":"Authentication successful"},"id":{"raw":"msg-12345"},"link":{"raw":"https://mail.example.com/message/12345"},"mailbox":{"raw":"user@example.com"},"protocol":{"raw":"IMAP"},"server":{"raw":"imap.example.com"},"source":{"raw":"81.2.69.144"},"time":"2019-10-21T18:23:24Z","zone_offset":"+00:00"},"received_spf":{"client_ip":{"ip":"1.128.0.0"},"description":"SPF record found","designator":"pass","helo":{"domain":"example.com","punycode":"example.com","root_domain":"example.com","sld":"example","subdomain":"example","tld":"com","valid":true},"server":{"domain":"mail.example.com","punycode":"mail.example.com","root_domain":"example.com","sld":"example","subdomain":"mail","tld":"com","valid":true},"verdict":"pass"},"signature":{"algorithm":"rsa-sha256","body_hash":"b9c4a3f9d93d9a38bdf8c47a8f2d2c79ec1d8b1f","domain":"example.com","headers":"from:to:subject:date","instance":"123456","selector":"default","signature":"d2abf9d6c8f4b8d68d8f3f7b6f9d3b8e6a8c2b3a9f4b8d7b9d3b6a8f9c3b4e5f","type":"spf","version":"1"}}],"in_reply_to":"in_reply_to_value","delivered_to":{"domain":{"domain":"example.com","subdomain":"example","tld":"com","email":"testing@sublimesecurity.com","punycode":"example.com","root_domain":"example.com","sld":"example","valid":true},"email":"testing@sublimesecurity.com","local_part":"testing"},"ips":[{"ip":"1.128.0.0"}],"mailer":"MyCustomMailer","message_id":"2fe271830bbad5fe3a70abbe7a8c0bfe7refe3ffe","domains":[{"domain":"test.com","subdomain":"test","tld":"com","punycode":"test.com","root_domain":"test.com","sld":"test","valid":true},{"domain":"example.com","subdomain":"example","tld":"com","punycode":"example.com","root_domain":"example.com","sld":"example","valid":true}],"reply_to":[{"email":{"email":"user@example.com","local_part":"user","domain":{"domain":"example.com","root_domain":"example.com","sld":"example","tld":"com","valid":true}}},{"display_name":"Example Display Name","email":{"domain":{"punycode":"example.com","subdomain":"sub.example"}}},{"display_name":"Another Display Name","email":{"domain":{"punycode":"anotherexample.com","subdomain":"sub.anotherexample"}}}]},"type":{"outbound":true},"mailbox":{"email":{"email":"user@example.com","local_part":"user","domain":{"domain":"example.com","root_domain":"example.com","sld":"example","tld":"com","valid":true,"punycode":"xn--example-d4a.com","subdomain":"sub"}}},"recipients":{"to":[{"display_name":"Alice Johnson","email":{"email":"user@example.com","local_part":"user","domain":{"domain":"example.com","root_domain":"example.com","sld":"example","tld":"com","valid":true,"punycode":"xn--example-d4a.net","subdomain":"sub"}}}],"bcc":[{"display_name":"John Doe","email":{"domain":{"domain":"example.com","punycode":"xn--example-d4a.com","root_domain":"example","sld":"example","subdomain":"sub","tld":"com","valid":true},"email":"john.doe@example.com","local_part":"john.doe"}}],"cc":[{"display_name":"Jane Smith","email":{"domain":{"domain":"example.org","punycode":"xn--example-d4a.org","root_domain":"example","sld":"example","subdomain":"sub","tld":"org","valid":true},"email":"jane.smith@example.org","local_part":"jane.smith"}}]},"sender":{"display_name":"Sublime Security Test","email":{"email":"testing@sublimesecurity.com","local_part":"testing","domain":{"domain":"sublimesecurity.com","root_domain":"sublimesecurity.com","sld":"sublimesecurity","tld":"com","valid":true,"punycode":"xn--example-d4a.com","subdomain":"sub"}}},"subject":{"subject":"Sublime-Security-Standard-Test-String"},"_meta":{"id":"01911208-633c-7f03-b303-e594d92cf818","canonical_id":"2fe271830bbad5fe3a70abbe7a8c0bfe79eb208a76cde267930d19f0e8cea81c","created_at":"2024-08-02T07:40:25.135939305Z","effective_at":"2024-08-02T07:40:25.135939305Z"},"_errors":[{"field":"Mime-Version","message":"No Mime-Version defined in headers","type":"missing_header_field"}]} diff --git a/packages/sublime_security/data_stream/email_message/_dev/deploy/tf/main.tf b/packages/sublime_security/data_stream/email_message/_dev/deploy/tf/main.tf new file mode 100644 index 00000000000..bf824481a06 --- /dev/null +++ b/packages/sublime_security/data_stream/email_message/_dev/deploy/tf/main.tf @@ -0,0 +1,57 @@ +provider "aws" { + region = "us-east-1" + default_tags { + tags = { + environment = var.ENVIRONMENT + repo = var.REPO + branch = var.BRANCH + build = var.BUILD_ID + created_date = var.CREATED_DATE + } + } +} + +resource "aws_s3_bucket" "bucket" { + bucket = "elastic-package-sublime-security-bucket-${var.TEST_RUN_ID}" +} + +resource "aws_sqs_queue" "queue" { + name = "elastic-package-sublime-security-queue-${var.TEST_RUN_ID}" + policy = <Sublime Security test message.

","inner_text":"

Sublime Security test message.

"},"ips":[{"ip":"1.128.0.0"}],"links":[{"display_text":"click here!","display_url":{"domain":{"domain":"example.com","punycode":"test","root_domain":"example.com","subdomain":"example","tld":"com","valid":true},"scheme":"https","fragment":"top","password":"pass123","path":"/search","port":80,"query_params":"q=elasticsearch","rewrite":{"encoders":["base64"],"original":"test"},"url":"https://example.com/search?q=elasticsearch#top","username":"user12"},"mismatched":true},{"href_url":{"domain":{"domain":"example.com","punycode":"test","root_domain":"example.com","subdomain":"example","tld":"com","valid":true},"scheme":"https","fragment":"top","password":"pass123","path":"/search","port":80,"query_params":"q=elasticsearch","rewrite":{"encoders":["base64"],"original":"test"},"url":"https://example.com/search?q=elasticsearch#top","username":"user12"}}]},"external":{"created_at":"2024-08-02T07:40:25.135939305Z","message_id":"2fe271830bbad5fe3a70abbe7a8c0bfe7refe3ffe","route_type":"sent","spam":false,"spam_folder":true},"attachments":[{"content_id":"abc123","content_transfer_encoding":"base64","content_type":"application/pdf","file_extension":".pdf","file_name":"sample_document.pdf","file_type":"document","md5":"1a2b3c","raw":"JVBERi0xLjMKJcfs4AAQSkZjRgABAQE","sha1":"4d5e6f","sha256":"7g8h9i","size":102400},{"content_id":"xyz456","content_transfer_encoding":"7bit","content_type":"image/jpeg","file_extension":".jpg","file_name":"image_photo.jpg","file_type":"image","md5":"7h8i9j","raw":"/9j/4AAQSkZJRgABAQEJVBERi0xLjMKJd","sha1":"1k2l3m","sha256":"4n5o6p","size":204800},{"content_id":"efg789","content_transfer_encoding":"quoted-printable","content_type":"text/plain","file_extension":".txt","file_name":"notes.txt","file_type":"text","md5":"1x2y3z","raw":"SGVsbG8gdVsbG8gd29yb29ybGQhVsbG8gd29yb","sha1":"4a5b6c","sha256":"7d8e9f","size":5120}],"headers":{"date":"2019-10-21T18:23:24Z","date_original_offset":"-4","hops":[{"index":0,"fields":[{"name":"To","value":"user@example.com","position":0},{"name":"Subject","value":"Sublime-Security-Standard-Test-String","position":1},{"name":"Date","value":"Mon, 21 Oct 2019 14:23:24 -0400","position":2},{"name":"From","value":"Sublime Security Test ","position":3}]}],"delivered_to":{"domain":{"domain":"example.com","subdomain":"example","tld":"com","email":"testing@sublimesecurity.com"}},"ips":[{"ip":"1.128.0.0"}],"mailer":"MyCustomMailer","message_id":"2fe271830bbad5fe3a70abbe7a8c0bfe7refe3ffe","domains":[{"domain":"test.com","subdomain":"test","tld":"com"},{"domain":"example.com","subdomain":"example","tld":"com"}],"reply_to":[{"email":{"email":"user@example.com","local_part":"user","domain":{"domain":"example.com","root_domain":"example.com","sld":"example","tld":"com","valid":true}}}]},"type":{"outbound":true},"mailbox":{"email":{"email":"user@example.com","local_part":"user","domain":{"domain":"example.com","root_domain":"example.com","sld":"example","tld":"com","valid":true}}},"recipients":{"to":[{"email":{"email":"user@example.com","local_part":"user","domain":{"domain":"example.com","root_domain":"example.com","sld":"example","tld":"com","valid":true}}}]},"sender":{"display_name":"Sublime Security Test","email":{"email":"testing@sublimesecurity.com","local_part":"testing","domain":{"domain":"sublimesecurity.com","root_domain":"sublimesecurity.com","sld":"sublimesecurity","tld":"com","valid":true}}},"subject":{"subject":"Sublime-Security-Standard-Test-String"},"_meta":{"id":"01911208-633c-7f03-b303-e594d92cf818","canonical_id":"2fe271830bbad5fe3a70abbe7a8c0bfe79eb208a76cde267930d19f0e8cea81c","created_at":"2024-08-02T07:40:25.135939305Z","effective_at":"2024-08-02T07:40:25.135939305Z"},"_errors":[{"field":"Mime-Version","message":"No Mime-Version defined in headers","type":"missing_header_field"}]} +{"body":{"plain":{"raw":"Sublime standard test message.\n"},"current_thread":{"text":"Sublime standard test message."}},"headers":{"date":"2019-10-21T18:23:24Z","date_original_offset":"-4","hops":[{"index":0,"fields":[{"name":"To","value":"user@example.com","position":0},{"name":"Subject","value":"Sublime-Standard-Test-String","position":1},{"name":"Date","value":"Mon, 21 Oct 2019 14:23:24 -0400","position":2},{"name":"From","value":"Sublime Test \u003ctesting@sublimesecurity.com\u003e","position":3}]}]},"type":{"inbound":true},"mailbox":{"email":{"email":"user@example.com","local_part":"user","domain":{"domain":"example.com","root_domain":"example.com","sld":"example","tld":"com","valid":true}}},"recipients":{"to":[{"email":{"email":"user@example.com","local_part":"user","domain":{"domain":"example.com","root_domain":"example.com","sld":"example","tld":"com","valid":true}}}]},"sender":{"display_name":"Sublime Test","email":{"email":"testing@sublimesecurity.com","local_part":"testing","domain":{"domain":"sublimesecurity.com","root_domain":"sublimesecurity.com","sld":"sublimesecurity","tld":"com","valid":true}}},"subject":{"subject":"Sublime-Standard-Test-String"},"_meta":{"id":"01912124-e085-7abc-aa36-5747852f7b42","canonical_id":"2fe271830bbad5fe3a70abbe7a8c0bfe79eb208a76cde267930d19f0e8cea81c","created_at":"2024-08-05T06:05:50.479598876Z","effective_at":"2024-08-05T06:05:50.479598876Z"},"_errors":[{"field":"Mime-Version","message":"No Mime-Version defined in headers","type":"missing_header_field"}]} +{"body":{"html":{"raw":"


---------- Forwarded message ---------
From: xyz Instructor: Alice <no-reply@e.example.com>
Date: Wed, 22 Nov, 2023, 3:07 pm
Subject: How to Achieve Your Career Goals in 2024
To: <alice123@test.com>


\r\n\r\n \r\n \r\n \r\n\r\n \r\n \r\n \r\n \r\n \r\n
\r\n \r\n \r\n \r\n","charset":"utf-8","content_transfer_encoding":"quoted-printable","inner_text":"---------- Forwarded message ---------\nFrom:\nxyz Instructor: Alice \n<\nno-reply@e.example.com\n>\nDate: Wed, 22 Nov, 2023, 3:07 pm\nSubject: How to Achieve Your Career Goals in 2024\nTo: <\nalice123@test.com\n>\nNew Educational Announcement\nHi Alice, an announcement has been made from Alice, instructor of\nLinux Administration: The Complete Linux Bootcamp for 2023\n.\n","display_text":"---------- Forwarded message ---------\nFrom: xyz Instructor: \nDate: Wed, 22 Nov, 2023, 3:07pm\nSubject: How to Achieve Your Career Goals in 2024\nTo: \nNew Educational Announcement\nHi Alice, an announcement has been made from Alice, instructor of Linux Administration: The Complete Linux Bootcamp for 2023."},"plain":{"raw":"---------- Forwarded message ---------\r\nFrom: xyz Instructor: Alice \r\nDate: Wed, 22 Nov, 2023, 3:07 pm\r\nSubject: How to Achieve Your Career Goals in 2024\r\nTo: \r\n\r\n\r\n\r\n\r\n* New Educational Announcement *\r\n\r\n","charset":"utf-8","content_transfer_encoding":"quoted-printable"},"links":[{"display_text":"Linux Administration: The Complete Linux Bootcamp for 2023","href_url":{"url":"https://e2.example.com/ls/click?upn=6n","domain":{"domain":"e2.example.com","root_domain":"example.com","sld":"example","subdomain":"e2","tld":"com","valid":true},"path":"/ls/click","query_params":"upn=Z","scheme":"https"}},{"display_text":"How I Went From Zero Job Offers To Working At xyz In 3 Years","href_url":{"url":"https://e2.example.com/ls/click?upn=6n","domain":{"domain":"e2.example.com","root_domain":"example.com","sld":"example","subdomain":"e2","tld":"com","valid":true},"path":"/ls/click","query_params":"upn=6n","scheme":"https"}},{"display_text":"How to beat Imposter Syndrome!","href_url":{"url":"https://e2.example.com/ls/click?upn=6nwDRRN3APL2A","domain":{"domain":"e2.example.com","root_domain":"example.com","sld":"example","subdomain":"e2","tld":"com","valid":true},"path":"/ls/click","query_params":"upn=6nwDRRN3APL2A","scheme":"https"}},{"display_text":"full state of AI for programmers","href_url":{"url":"https://e2.example.com/ls/click?upn=6nwDR","domain":{"domain":"e2.example.com","root_domain":"example.com","sld":"example","subdomain":"e2","tld":"com","valid":true},"path":"/ls/click","query_params":"upn=6nwDR","scheme":"https"}},{"display_text":"See Announcement","href_url":{"url":"https://e2.example.com/ls/click?upn=ZF3sOyS2SxEPIoSZT6Aoc","domain":{"domain":"e2.example.com","root_domain":"example.com","sld":"example","subdomain":"e2","tld":"com","valid":true},"path":"/ls/click","query_params":"upn=ZF3sOyS2SxEPIoSZT6Aoc","scheme":"https"}},{"display_text":"unsubscribe","href_url":{"url":"https://e2.example.com/ls/click?upn=ZF3sOyS2S","domain":{"domain":"e2.example.com","root_domain":"example.com","sld":"example","subdomain":"e2","tld":"com","valid":true},"path":"/ls/click","query_params":"upn=ZF3sOyS2S","scheme":"https"}},{"display_text":"report abuse","href_url":{"url":"https://e2.example.com/ls/click?upn=ZF3s","domain":{"domain":"e2.example.com","root_domain":"example.com","sld":"example","subdomain":"e2","tld":"com","valid":true},"path":"/ls/click","query_params":"upn=ZF3s","scheme":"https"}},{"display_url":{"url":"e.example.com","domain":{"domain":"e.example.com","root_domain":"example.com","sld":"example","subdomain":"e","tld":"com","valid":true},"scheme":"http"},"href_url":{"url":"e.example.com","domain":{"domain":"e.example.com","root_domain":"example.com","sld":"example","subdomain":"e","tld":"com","valid":true},"scheme":"http"}},{"display_url":{"url":"test.com","domain":{"domain":"test.com","root_domain":"test.com","sld":"test","tld":"com","valid":true},"scheme":"http"},"href_url":{"url":"test.com","domain":{"domain":"test.com","root_domain":"test.com","sld":"test","tld":"com","valid":true},"scheme":"http"}}],"current_thread":{"text":""}},"external":{"created_at":"2024-08-08T06:39:47Z","message_id":"11","route_type":"received","spam":false},"headers":{"auth_summary":{"dmarc":{"pass":true,"received_hop":0,"details":{"version":null,"verdict":"pass","action":null,"policy":"NONE","sub_policy":"QUARANTINE","disposition":"NONE","from":{"domain":"test.com","root_domain":"test.com","sld":"test","tld":"com","valid":true}}},"spf":{"pass":true,"error":false,"received_hop":0,"details":{"verdict":"pass","server":{"domain":"test.com","root_domain":"test.com","sld":"test","tld":"com","valid":true},"client_ip":{"ip":"81.2.69.192"},"designator":"domain of alice123@test.com","description":"test.com: domain of alice123@test.com designates 81.2.69.192 as permitted sender"}}},"date":"2024-08-08T06:39:33Z","date_original_offset":"5","delivered_to":{"email":"john123@test.com","local_part":"john123","domain":{"domain":"test.com","root_domain":"test.com","sld":"test","tld":"com","valid":true}},"domains":[{"domain":"mail-sor-f41.test.com","root_domain":"test.com","sld":"test","subdomain":"mail-sor-f41","tld":"com","valid":true},{"domain":"mx.test.com","root_domain":"test.com","sld":"test","subdomain":"mx","tld":"com","valid":true},{"domain":"test.com","root_domain":"test.com","sld":"test","tld":"com","valid":true}],"hops":[{"index":0,"authentication_results":{"compauth": { "verdict": "pass", "reason": "reason_value" },"type":"standard","dkim":"pass","dkim_details":[{"type":"dkim","instance":"@test.com","selector":"20230601","signature":"elrBA+fb"}],"dmarc":"pass","dmarc_details":{"version":null,"verdict":"pass","action":null,"policy":"NONE","sub_policy":"QUARANTINE","disposition":"NONE","from":{"domain":"test.com","root_domain":"test.com","sld":"test","tld":"com","valid":true}},"spf":"pass","spf_details":{"verdict":"pass","server":{"domain":"test.com","root_domain":"test.com","sld":"test","tld":"com","valid":true},"client_ip":{"ip":"81.2.69.192"},"designator":"alice123@test.com","description":"test.com: domain of alice123@test.com designates 81.2.69.192 as permitted sender"},"server":{"domain":"mx.test.com","root_domain":"test.com","sld":"test","subdomain":"mx","tld":"com","valid":true}},"signature":{"type":"dkim","version":"1","algorithm":"rsa-sha256","selector":"20230601","signature":"elrBA+fbKpLqfjEOj7s4cAR9f8s8Tz0lHFsCjgE4t5dSXKpL6YZRZ44EPNZ1I1ROJx jTmQ/zz41IUs7K4tWqtuJnM98GSCsEvc31Lr3w99k/kjWm0Raei5dWc/4OqxZKmEiV3J r4Un6IlHY5CiUO","body_hash":"taAa59+CAu/Cmad8+7VJw9KpMlCezg5RHDG7X7f91P4=","domain":"test.com","headers":"to:subject:message-id:date:from:in-reply-to:references:mime-version :from:to:cc:subject:date:message-id:reply-to"},"received_spf":{"verdict":"pass","server":{"domain":"test.com","root_domain":"test.com","sld":"test","tld":"com","valid":true},"client_ip":{"ip":"81.2.69.192"},"designator":"domain of alice123@test.com","description":"test.com: domain of alice123@test.com designates 81.2.69.192 as permitted sender"},"fields":[{"name":"Content-Type","value":"multipart/alternative; boundary=\"000000000000e2ee4a061f264a8a\"","position":0},{"name":"To","value":"john123@test.com","position":1},{"name":"Subject","value":"Fwd: How to Achieve Your Career Goals in 2024","position":2},{"name":"Message-ID","value":"","position":3},{"name":"Date","value":"Thu, 8 Aug 2024 12:09:33 +0530","position":4},{"name":"From","value":"Alice ","position":5},{"name":"In-Reply-To","value":"","position":6},{"name":"References","value":"","position":7},{"name":"MIME-Version","value":"1.0","position":8},{"name":"X-Received","value":"by 2002:a05:6512:12cc:b0:52b:faa1:7c74 with SMTP id 2adb3069b0e04-530e5d21173mr181491e87.5.1723099185866; Wed, 07 Aug 2024 23:39:45 -0700 (PDT)","position":9},{"name":"X-Test-Smtp-Source","value":"AGHT+IEOPD6/KQ1QGxfpqSsNfRVL3tyvz5l2ZkiY/swtxVoJf6PjrsQMELt982FDdWiYdCh6nHFEnpPzh50Rx+gpG+8=","position":10},{"name":"X-Gm-Message-State","value":"AOJu0YyLnr/Zjgr7uXbT4OWJKXOdcruE0DB5eYAxH2GMsUDqfM81HEem KMt/4qXMzEYtzJJ+2fkjA2zpg2e3zN9iqNuiUI86WAOTl07NfeqPMJCIDDz0Q0gfeiOwIK8rlL8 jammp+/G9xCHMayr+inDiqemweFRyvA==","position":11},{"name":"X-Test-DKIM-Signature","value":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723099186; x=1723703986; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=taAa59+CAu/Cmad8+7VJw9KpMlCezg5RHDG7X7f91P4=; b=VcRvW5nfz7WbzaIuyv5g5x/Z2U0qbrDR6qP","position":12},{"name":"DKIM-Signature","value":"v=1; a=rsa-sha256; c=relaxed/relaxed; d=test.com; s=20230601; t=1723099186; x=1723703986; dara=test.com; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :from:to:cc:subject:date:message-id:reply-to; bh=taAa59+CAu/Cmad8+7VJw9KpMlCezg5RHDG7X7f91P4=","position":13},{"name":"Authentication-Results","value":"mx.test.com; dkim=pass header.i=@test.com header.s=20230601 header.b=elrBA+fb; spf=pass (test.com: domain of alice123@test.com designates 81.2.69.192 as permitted sender) smtp.mailfrom=alice123@test.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=test.com; dara=pass header.i=@test.com","position":14},{"name":"Received-SPF","value":"pass (test.com: domain of alice123@test.com designates 81.2.69.192 as permitted sender) client-ip=81.2.69.192;","position":15},{"name":"Received","value":"from mail-sor-f41.test.com (mail-sor-f41.test.com. [81.2.69.192]) by mx.test.com with SMTPS id 2adb3069b0e04-530de79d133sor508979e87.12.2024.08.07.23.39.46 for (Test Transport Security); Wed, 07 Aug 2024 23:39:47 -0700 (PDT)","position":16}],"received":{"source":{"raw":"mail-sor-f41.test.com (mail-sor-f41.test.com. [81.2.69.192])"},"server":{"raw":"mx.test.com"},"mailbox":{"raw":""},"protocol":{"raw":"SMTPS"},"id":{"raw":"2adb3069b1e04-530de79d133sor508979e87.12.2024.08.07.23.39.46"},"additional":{"raw":"(Test Transport Security)"},"time":"2024-08-08T06:39:47Z","zone_offset":"-7"}},{"index":1,"authentication_results":{"type":"arc","instance":"1","dkim":"pass","dkim_details":[{"type":"dkim","instance":"@test.com","selector":"20230601","signature":"elrBA+fb"}],"dmarc":"pass","dmarc_details":{"version":null,"verdict":"pass","action":null,"policy":"NONE","sub_policy":"QUARANTINE","disposition":"NONE","from":{"domain":"test.com","root_domain":"test.com","sld":"test","tld":"com","valid":true}},"spf":"pass","spf_details":{"verdict":"pass","server":{"domain":"test.com","root_domain":"test.com","sld":"test","tld":"com","valid":true},"client_ip":{"ip":"81.2.69.192"},"designator":"alice123@test.com","description":"test.com: domain of alice123@test.com designates 81.2.69.192 as permitted sender"},"server":{"domain":"mx.test.com","root_domain":"test.com","sld":"test","subdomain":"mx","tld":"com","valid":true}},"signature":{"type":"arc-message","instance":"1","algorithm":"rsa-sha256","selector":"arc-20160816","signature":"nKqpy2hvLAXWHwdm39Mg1dL6lziVFqVY7ikY9FaP1w0pDHO6t0zbiMwcwSkS/Crz+ Y38+/FHiPhk65AocA0Yzw9P96RpK60iDaHfXpEBsxJIhJt9GN7","body_hash":"taAa59+CAu/Cmad8+7VJw9KpMlCezg5RHDG7X7f91P4=","domain":"test.com","headers":"to:subject:message-id:date:from:in-reply-to:references:mime-version :dkim-signature"},"fields":[{"name":"Return-Path","value":"","position":17},{"name":"ARC-Authentication-Results","value":"i=1; mx.test.com; dkim=pass header.i=@test.com header.s=20230601 header.b=elrBA+fb; spf=pass (test.com: domain of alice123@test.com designates 81.2.69.192 as permitted sender) smtp.mailfrom=alice123@test.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=test.com; dara=pass header.i=@test.com","position":18},{"name":"ARC-Message-Signature","value":"i=1; a=rsa-sha256; c=relaxed/relaxed; d=test.com; s=arc-20160816; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :dkim-signature; bh=taAa59+CAu/Cmad8+7VJw9KpMlCezg5RHDG7X7f91P4=; fh=rEZ8N7lZwF+f2DJz7PZfSiLZqwmiZLvrdguxWR5M0Mw=; b=nKqpy2hvLAXWHw39Mg1dLY6lziVFqVY7ikY9FaP1","position":19},{"name":"ARC-Seal","value":"i=1; a=rsa-sha256; t=1723099187; cv=none; d=test.com; s=arc-20160816; b=s4+J1/60S5sNdJ0Fd56rNghLRYU+m7QHad7No6E0iBi+7WGCuOOd2w07CSfEx++0jx Y0lBuDGDzNrGGHVpfi3ODGvx/aoU2vg8/siNaHAnIR4ADSbV+sr67vFiIEPqYNmQyihC lhOm0gnxDVD7ozZ","position":20},{"name":"X-Received","value":"by 2002:a05:6512:1242:b0:52f:c398:8780 with SMTP id 2adb3069b0e04-530e5d70e42mr184435e87.18.1723099187103; Wed, 07 Aug 2024 23:39:47 -0700 (PDT)","position":21},{"name":"Received","value":"by 2002:a05:6850:988a:b0:5bb:ddaf:ae20 with SMTP id li10csp662340nnb; Wed, 7 Aug 2024 23:39:47 -0700 (PDT)","position":22}],"received":{"server":{"raw":"2002:a05:6850:988a:b0:5bb:ddaf:ae20"},"protocol":{"raw":"SMTP"},"id":{"raw":"li10csp662340nnb"},"time":"2024-08-08T06:39:47Z","zone_offset":"-7"}},{"index":2,"fields":[{"name":"Delivered-To","value":"john123@test.com","position":23}]}],"ips":[{"ip":"81.2.69.192"}],"message_id":"","references":["hpfhx9h8QtSRnWCE_AzviQ@geopod-ismtpd-56"],"in_reply_to":"","return_path":{"email":"alice123@test.com","local_part":"alice123","domain":{"domain":"test.com","root_domain":"test.com","sld":"test","tld":"com","valid":true}}},"type":{"inbound":true},"mailbox":{"email":{"email":"john123@test.com","local_part":"john123","domain":{"domain":"test.com","root_domain":"test.com","sld":"test","tld":"com","valid":true}}},"recipients":{"to":[{"email":{"email":"john123@test.com","local_part":"john123","domain":{"domain":"test.com","root_domain":"test.com","sld":"test","tld":"com","valid":true}}}]},"sender":{"display_name":"Alice","email":{"email":"alice123@test.com","local_part":"alice123","domain":{"domain":"test.com","root_domain":"test.com","sld":"test","tld":"com","valid":true}}},"subject":{"subject":"Fwd: How to Achieve Your Career Goals in 2024"},"_meta":{"id":"019130be-779c-7641-87c7-284e8dcb10e5","canonical_id":"dc767a9c58a14ea5560b5786c644d5cb9ef7cadb11f680231ec09e8fdfeb4d53","created_at":"2024-08-08T06:47:54.460399216Z","effective_at":"2024-08-08T06:39:47Z"}} diff --git a/packages/sublime_security/data_stream/email_message/_dev/test/pipeline/test-email-message.log-expected.json b/packages/sublime_security/data_stream/email_message/_dev/test/pipeline/test-email-message.log-expected.json new file mode 100644 index 00000000000..a91ef4594d7 --- /dev/null +++ b/packages/sublime_security/data_stream/email_message/_dev/test/pipeline/test-email-message.log-expected.json @@ -0,0 +1,1419 @@ +{ + "expected": [ + { + "@timestamp": "2024-08-02T07:40:25.135Z", + "destination": { + "domain": "example.com", + "subdomain": "example", + "top_level_domain": "com" + }, + "ecs": { + "version": "8.11.0" + }, + "email": { + "attachments": [ + { + "file": { + "extension": "pdf", + "hash": { + "md5": "1a2b3c", + "sha1": "4d5e6f", + "sha256": "7g8h9i" + }, + "mime_type": "application/pdf", + "name": "sample_document.pdf", + "size": 102400 + } + }, + { + "file": { + "extension": "jpg", + "hash": { + "md5": "7h8i9j", + "sha1": "1k2l3m", + "sha256": "4n5o6p" + }, + "mime_type": "image/jpeg", + "name": "image_photo.jpg", + "size": 204800 + } + }, + { + "file": { + "extension": "txt", + "hash": { + "md5": "1x2y3z", + "sha1": "4a5b6c", + "sha256": "7d8e9f" + }, + "mime_type": "text/plain", + "name": "notes.txt", + "size": 5120 + } + } + ], + "direction": "outbound", + "from": { + "address": [ + "testing@sublimesecurity.com" + ] + }, + "message_id": "2fe271830bbad5fe3a70abbe7a8c0bfe7refe3ffe", + "origination_timestamp": "2024-08-02T07:40:25.135Z", + "reply_to": { + "address": [ + "user@example.com" + ] + }, + "subject": "Sublime-Security-Standard-Test-String", + "to": { + "address": [ + "user@example.com" + ] + }, + "x_mailer": "MyCustomMailer" + }, + "event": { + "category": [ + "email" + ], + "id": "01911208-633c-7f03-b303-e594d92cf818", + "kind": "event", + "original": "{\"body\":{\"plain\":{\"raw\":\"Sublime Security test message.\\n\"},\"current_thread\":{\"text\":\"Sublime Security test message.\"},\"html\":{\"charset\":\"utf-8\",\"content_transfer_encoding\":\"base64\",\"display_text\":\"Sublime Security test message.\",\"raw\":\"

Sublime Security test message.

\",\"inner_text\":\"

Sublime Security test message.

\"},\"ips\":[{\"ip\":\"1.128.0.0\"}],\"links\":[{\"display_text\":\"click here!\",\"display_url\":{\"domain\":{\"domain\":\"example.com\",\"punycode\":\"test\",\"root_domain\":\"example.com\",\"subdomain\":\"example\",\"tld\":\"com\",\"valid\":true},\"scheme\":\"https\",\"fragment\":\"top\",\"password\":\"pass123\",\"path\":\"/search\",\"port\":80,\"query_params\":\"q=elasticsearch\",\"rewrite\":{\"encoders\":[\"base64\"],\"original\":\"test\"},\"url\":\"https://example.com/search?q=elasticsearch#top\",\"username\":\"user12\"},\"mismatched\":true},{\"href_url\":{\"domain\":{\"domain\":\"example.com\",\"punycode\":\"test\",\"root_domain\":\"example.com\",\"subdomain\":\"example\",\"tld\":\"com\",\"valid\":true},\"scheme\":\"https\",\"fragment\":\"top\",\"password\":\"pass123\",\"path\":\"/search\",\"port\":80,\"query_params\":\"q=elasticsearch\",\"rewrite\":{\"encoders\":[\"base64\"],\"original\":\"test\"},\"url\":\"https://example.com/search?q=elasticsearch#top\",\"username\":\"user12\"}}]},\"external\":{\"created_at\":\"2024-08-02T07:40:25.135939305Z\",\"message_id\":\"2fe271830bbad5fe3a70abbe7a8c0bfe7refe3ffe\",\"route_type\":\"sent\",\"spam\":false,\"spam_folder\":true},\"attachments\":[{\"content_id\":\"abc123\",\"content_transfer_encoding\":\"base64\",\"content_type\":\"application/pdf\",\"file_extension\":\".pdf\",\"file_name\":\"sample_document.pdf\",\"file_type\":\"document\",\"md5\":\"1a2b3c\",\"raw\":\"JVBERi0xLjMKJcfs4AAQSkZjRgABAQE\",\"sha1\":\"4d5e6f\",\"sha256\":\"7g8h9i\",\"size\":102400},{\"content_id\":\"xyz456\",\"content_transfer_encoding\":\"7bit\",\"content_type\":\"image/jpeg\",\"file_extension\":\".jpg\",\"file_name\":\"image_photo.jpg\",\"file_type\":\"image\",\"md5\":\"7h8i9j\",\"raw\":\"/9j/4AAQSkZJRgABAQEJVBERi0xLjMKJd\",\"sha1\":\"1k2l3m\",\"sha256\":\"4n5o6p\",\"size\":204800},{\"content_id\":\"efg789\",\"content_transfer_encoding\":\"quoted-printable\",\"content_type\":\"text/plain\",\"file_extension\":\".txt\",\"file_name\":\"notes.txt\",\"file_type\":\"text\",\"md5\":\"1x2y3z\",\"raw\":\"SGVsbG8gdVsbG8gd29yb29ybGQhVsbG8gd29yb\",\"sha1\":\"4a5b6c\",\"sha256\":\"7d8e9f\",\"size\":5120}],\"headers\":{\"date\":\"2019-10-21T18:23:24Z\",\"date_original_offset\":\"-4\",\"hops\":[{\"index\":0,\"fields\":[{\"name\":\"To\",\"value\":\"user@example.com\",\"position\":0},{\"name\":\"Subject\",\"value\":\"Sublime-Security-Standard-Test-String\",\"position\":1},{\"name\":\"Date\",\"value\":\"Mon, 21 Oct 2019 14:23:24 -0400\",\"position\":2},{\"name\":\"From\",\"value\":\"Sublime Security Test \",\"position\":3}]}],\"delivered_to\":{\"domain\":{\"domain\":\"example.com\",\"subdomain\":\"example\",\"tld\":\"com\",\"email\":\"testing@sublimesecurity.com\"}},\"ips\":[{\"ip\":\"1.128.0.0\"}],\"mailer\":\"MyCustomMailer\",\"message_id\":\"2fe271830bbad5fe3a70abbe7a8c0bfe7refe3ffe\",\"domains\":[{\"domain\":\"test.com\",\"subdomain\":\"test\",\"tld\":\"com\"},{\"domain\":\"example.com\",\"subdomain\":\"example\",\"tld\":\"com\"}],\"reply_to\":[{\"email\":{\"email\":\"user@example.com\",\"local_part\":\"user\",\"domain\":{\"domain\":\"example.com\",\"root_domain\":\"example.com\",\"sld\":\"example\",\"tld\":\"com\",\"valid\":true}}}]},\"type\":{\"outbound\":true},\"mailbox\":{\"email\":{\"email\":\"user@example.com\",\"local_part\":\"user\",\"domain\":{\"domain\":\"example.com\",\"root_domain\":\"example.com\",\"sld\":\"example\",\"tld\":\"com\",\"valid\":true}}},\"recipients\":{\"to\":[{\"email\":{\"email\":\"user@example.com\",\"local_part\":\"user\",\"domain\":{\"domain\":\"example.com\",\"root_domain\":\"example.com\",\"sld\":\"example\",\"tld\":\"com\",\"valid\":true}}}]},\"sender\":{\"display_name\":\"Sublime Security Test\",\"email\":{\"email\":\"testing@sublimesecurity.com\",\"local_part\":\"testing\",\"domain\":{\"domain\":\"sublimesecurity.com\",\"root_domain\":\"sublimesecurity.com\",\"sld\":\"sublimesecurity\",\"tld\":\"com\",\"valid\":true}}},\"subject\":{\"subject\":\"Sublime-Security-Standard-Test-String\"},\"_meta\":{\"id\":\"01911208-633c-7f03-b303-e594d92cf818\",\"canonical_id\":\"2fe271830bbad5fe3a70abbe7a8c0bfe79eb208a76cde267930d19f0e8cea81c\",\"created_at\":\"2024-08-02T07:40:25.135939305Z\",\"effective_at\":\"2024-08-02T07:40:25.135939305Z\"},\"_errors\":[{\"field\":\"Mime-Version\",\"message\":\"No Mime-Version defined in headers\",\"type\":\"missing_header_field\"}]}", + "type": [ + "info" + ] + }, + "observer": { + "product": "Sublime Security", + "vendor": "Sublime Security" + }, + "related": { + "hash": [ + "1a2b3c", + "7h8i9j", + "1x2y3z", + "4d5e6f", + "1k2l3m", + "4a5b6c", + "7g8h9i", + "4n5o6p", + "7d8e9f" + ], + "hosts": [ + "example.com", + "test.com", + "sublimesecurity.com" + ], + "ip": [ + "1.128.0.0" + ], + "user": [ + "user12", + "user@example.com", + "testing@sublimesecurity.com" + ] + }, + "source": { + "domain": "sublimesecurity.com", + "top_level_domain": "com" + }, + "sublime_security": { + "email_message": { + "attachments": [ + { + "content": { + "id": "abc123", + "transfer_encoding": "base64", + "type": "application/pdf" + }, + "file": { + "extension": ".pdf", + "name": "sample_document.pdf", + "type": "document" + }, + "md5": "1a2b3c", + "raw": "JVBERi0xLjMKJcfs4AAQSkZjRgABAQE", + "sha1": "4d5e6f", + "sha256": "7g8h9i", + "size": 102400 + }, + { + "content": { + "id": "xyz456", + "transfer_encoding": "7bit", + "type": "image/jpeg" + }, + "file": { + "extension": ".jpg", + "name": "image_photo.jpg", + "type": "image" + }, + "md5": "7h8i9j", + "raw": "/9j/4AAQSkZJRgABAQEJVBERi0xLjMKJd", + "sha1": "1k2l3m", + "sha256": "4n5o6p", + "size": 204800 + }, + { + "content": { + "id": "efg789", + "transfer_encoding": "quoted-printable", + "type": "text/plain" + }, + "file": { + "extension": ".txt", + "name": "notes.txt", + "type": "text" + }, + "md5": "1x2y3z", + "raw": "SGVsbG8gdVsbG8gd29yb29ybGQhVsbG8gd29yb", + "sha1": "4a5b6c", + "sha256": "7d8e9f", + "size": 5120 + } + ], + "body": { + "current_thread": { + "text": "Sublime Security test message." + }, + "html": { + "charset": "utf-8", + "content_transfer_encoding": "base64", + "display_text": "Sublime Security test message.", + "inner_text": "

Sublime Security test message.

", + "raw": "

Sublime Security test message.

" + }, + "ips": [ + { + "ip": "1.128.0.0" + } + ], + "links": [ + { + "display_text": "click here!", + "display_url": { + "domain": { + "domain": "example.com", + "punycode": "test", + "root_domain": "example.com", + "subdomain": "example", + "tld": "com", + "valid": true + }, + "fragment": "top", + "password": "pass123", + "path": "/search", + "port": 80, + "query_params": "q=elasticsearch", + "rewrite": { + "encoders": [ + "base64" + ], + "original": "test" + }, + "scheme": "https", + "url": "https://example.com/search?q=elasticsearch#top", + "username": "user12" + }, + "mismatched": true + }, + { + "href_url": { + "domain": { + "domain": "example.com", + "punycode": "test", + "root_domain": "example.com", + "subdomain": "example", + "tld": "com", + "valid": true + }, + "fragment": "top", + "password": "pass123", + "path": "/search", + "port": 80, + "query_params": "q=elasticsearch", + "rewrite": { + "encoders": [ + "base64" + ], + "original": "test" + }, + "scheme": "https", + "url": "https://example.com/search?q=elasticsearch#top", + "username": "user12" + } + } + ], + "plain": { + "raw": "Sublime Security test message.\n" + } + }, + "errors": [ + { + "field": "Mime-Version", + "message": "No Mime-Version defined in headers", + "type": "missing_header_field" + } + ], + "external": { + "created_at": "2024-08-02T07:40:25.135Z", + "message_id": "2fe271830bbad5fe3a70abbe7a8c0bfe7refe3ffe", + "route_type": "sent", + "spam": false, + "spam_folder": true + }, + "headers": { + "date": "2019-10-21T18:23:24.000Z", + "date_original_offset": "-4", + "delivered_to": { + "domain": { + "domain": "example.com", + "subdomain": "example", + "tld": "com" + } + }, + "domains": [ + { + "domain": "test.com", + "subdomain": "test", + "tld": "com" + }, + { + "domain": "example.com", + "subdomain": "example", + "tld": "com" + } + ], + "hops": [ + { + "fields": [ + { + "name": "To", + "position": 0, + "to": "user@example.com", + "value": "user@example.com" + }, + { + "name": "Subject", + "position": 1, + "subject": "Sublime-Security-Standard-Test-String", + "value": "Sublime-Security-Standard-Test-String" + }, + { + "date": "Mon, 21 Oct 2019 14:23:24 -0400", + "name": "Date", + "position": 2, + "value": "Mon, 21 Oct 2019 14:23:24 -0400" + }, + { + "from": "Sublime Security Test ", + "name": "From", + "position": 3, + "value": "Sublime Security Test " + } + ], + "index": 0 + } + ], + "ips": [ + { + "ip": "1.128.0.0" + } + ], + "mailer": "MyCustomMailer", + "message_id": "2fe271830bbad5fe3a70abbe7a8c0bfe7refe3ffe", + "reply_to": [ + { + "email": { + "domain": { + "domain": "example.com", + "root_domain": "example.com", + "sld": "example", + "tld": "com", + "valid": true + }, + "local_part": "user", + "value": "user@example.com" + } + } + ] + }, + "mailbox": { + "email": { + "domain": { + "domain": "example.com", + "root_domain": "example.com", + "sld": "example", + "tld": "com", + "valid": true + }, + "local_part": "user", + "value": "user@example.com" + } + }, + "meta": { + "canonical_id": "2fe271830bbad5fe3a70abbe7a8c0bfe79eb208a76cde267930d19f0e8cea81c", + "created_at": "2024-08-02T07:40:25.135Z", + "effective_at": "2024-08-02T07:40:25.135Z", + "id": "01911208-633c-7f03-b303-e594d92cf818" + }, + "recipients": { + "to": [ + { + "email": { + "domain": { + "domain": "example.com", + "root_domain": "example.com", + "sld": "example", + "tld": "com", + "valid": true + }, + "local_part": "user", + "value": "user@example.com" + } + } + ] + }, + "sender": { + "display_name": "Sublime Security Test", + "email": { + "domain": { + "domain": "sublimesecurity.com", + "root_domain": "sublimesecurity.com", + "sld": "sublimesecurity", + "tld": "com", + "valid": true + }, + "local_part": "testing", + "value": "testing@sublimesecurity.com" + } + }, + "subject": { + "subject": "Sublime-Security-Standard-Test-String" + }, + "type": { + "outbound": true + } + } + }, + "tags": [ + "preserve_duplicate_custom_fields" + ], + "url": [ + { + "domain": "example.com", + "fragment": "top", + "full": "https://example.com/search?q=elasticsearch#top", + "password": "pass123", + "path": "/search", + "port": 80, + "query": "q=elasticsearch", + "scheme": "https", + "subdomain": "example", + "top_level_domain": "com", + "username": "user12" + } + ], + "user_agent": { + "device": { + "name": "Other" + }, + "name": "Other", + "original": "MyCustomMailer" + } + }, + { + "@timestamp": "2024-08-05T06:05:50.479Z", + "ecs": { + "version": "8.11.0" + }, + "email": { + "direction": "inbound", + "from": { + "address": [ + "testing@sublimesecurity.com" + ] + }, + "subject": "Sublime-Standard-Test-String", + "to": { + "address": [ + "user@example.com" + ] + } + }, + "event": { + "category": [ + "email" + ], + "id": "01912124-e085-7abc-aa36-5747852f7b42", + "kind": "event", + "original": "{\"body\":{\"plain\":{\"raw\":\"Sublime standard test message.\\n\"},\"current_thread\":{\"text\":\"Sublime standard test message.\"}},\"headers\":{\"date\":\"2019-10-21T18:23:24Z\",\"date_original_offset\":\"-4\",\"hops\":[{\"index\":0,\"fields\":[{\"name\":\"To\",\"value\":\"user@example.com\",\"position\":0},{\"name\":\"Subject\",\"value\":\"Sublime-Standard-Test-String\",\"position\":1},{\"name\":\"Date\",\"value\":\"Mon, 21 Oct 2019 14:23:24 -0400\",\"position\":2},{\"name\":\"From\",\"value\":\"Sublime Test \\u003ctesting@sublimesecurity.com\\u003e\",\"position\":3}]}]},\"type\":{\"inbound\":true},\"mailbox\":{\"email\":{\"email\":\"user@example.com\",\"local_part\":\"user\",\"domain\":{\"domain\":\"example.com\",\"root_domain\":\"example.com\",\"sld\":\"example\",\"tld\":\"com\",\"valid\":true}}},\"recipients\":{\"to\":[{\"email\":{\"email\":\"user@example.com\",\"local_part\":\"user\",\"domain\":{\"domain\":\"example.com\",\"root_domain\":\"example.com\",\"sld\":\"example\",\"tld\":\"com\",\"valid\":true}}}]},\"sender\":{\"display_name\":\"Sublime Test\",\"email\":{\"email\":\"testing@sublimesecurity.com\",\"local_part\":\"testing\",\"domain\":{\"domain\":\"sublimesecurity.com\",\"root_domain\":\"sublimesecurity.com\",\"sld\":\"sublimesecurity\",\"tld\":\"com\",\"valid\":true}}},\"subject\":{\"subject\":\"Sublime-Standard-Test-String\"},\"_meta\":{\"id\":\"01912124-e085-7abc-aa36-5747852f7b42\",\"canonical_id\":\"2fe271830bbad5fe3a70abbe7a8c0bfe79eb208a76cde267930d19f0e8cea81c\",\"created_at\":\"2024-08-05T06:05:50.479598876Z\",\"effective_at\":\"2024-08-05T06:05:50.479598876Z\"},\"_errors\":[{\"field\":\"Mime-Version\",\"message\":\"No Mime-Version defined in headers\",\"type\":\"missing_header_field\"}]}", + "type": [ + "info" + ] + }, + "observer": { + "product": "Sublime Security", + "vendor": "Sublime Security" + }, + "related": { + "hosts": [ + "example.com", + "sublimesecurity.com" + ], + "user": [ + "user@example.com", + "testing@sublimesecurity.com" + ] + }, + "source": { + "domain": "sublimesecurity.com", + "top_level_domain": "com" + }, + "sublime_security": { + "email_message": { + "body": { + "current_thread": { + "text": "Sublime standard test message." + }, + "plain": { + "raw": "Sublime standard test message.\n" + } + }, + "errors": [ + { + "field": "Mime-Version", + "message": "No Mime-Version defined in headers", + "type": "missing_header_field" + } + ], + "headers": { + "date": "2019-10-21T18:23:24.000Z", + "date_original_offset": "-4", + "hops": [ + { + "fields": [ + { + "name": "To", + "position": 0, + "to": "user@example.com", + "value": "user@example.com" + }, + { + "name": "Subject", + "position": 1, + "subject": "Sublime-Standard-Test-String", + "value": "Sublime-Standard-Test-String" + }, + { + "date": "Mon, 21 Oct 2019 14:23:24 -0400", + "name": "Date", + "position": 2, + "value": "Mon, 21 Oct 2019 14:23:24 -0400" + }, + { + "from": "Sublime Test ", + "name": "From", + "position": 3, + "value": "Sublime Test " + } + ], + "index": 0 + } + ] + }, + "mailbox": { + "email": { + "domain": { + "domain": "example.com", + "root_domain": "example.com", + "sld": "example", + "tld": "com", + "valid": true + }, + "local_part": "user", + "value": "user@example.com" + } + }, + "meta": { + "canonical_id": "2fe271830bbad5fe3a70abbe7a8c0bfe79eb208a76cde267930d19f0e8cea81c", + "created_at": "2024-08-05T06:05:50.479Z", + "effective_at": "2024-08-05T06:05:50.479Z", + "id": "01912124-e085-7abc-aa36-5747852f7b42" + }, + "recipients": { + "to": [ + { + "email": { + "domain": { + "domain": "example.com", + "root_domain": "example.com", + "sld": "example", + "tld": "com", + "valid": true + }, + "local_part": "user", + "value": "user@example.com" + } + } + ] + }, + "sender": { + "display_name": "Sublime Test", + "email": { + "domain": { + "domain": "sublimesecurity.com", + "root_domain": "sublimesecurity.com", + "sld": "sublimesecurity", + "tld": "com", + "valid": true + }, + "local_part": "testing", + "value": "testing@sublimesecurity.com" + } + }, + "subject": { + "subject": "Sublime-Standard-Test-String" + }, + "type": { + "inbound": true + } + } + }, + "tags": [ + "preserve_duplicate_custom_fields" + ] + }, + { + "@timestamp": "2024-08-08T06:47:54.460Z", + "destination": { + "domain": "test.com", + "top_level_domain": "com" + }, + "ecs": { + "version": "8.11.0" + }, + "email": { + "direction": "inbound", + "from": { + "address": [ + "alice123@test.com" + ] + }, + "message_id": "", + "origination_timestamp": "2024-08-08T06:39:47.000Z", + "subject": "Fwd: How to Achieve Your Career Goals in 2024", + "to": { + "address": [ + "john123@test.com" + ] + } + }, + "event": { + "category": [ + "email" + ], + "id": "019130be-779c-7641-87c7-284e8dcb10e5", + "kind": "event", + "original": "{\"body\":{\"html\":{\"raw\":\"


---------- Forwarded message ---------
From: xyz Instructor: Alice <no-reply@e.example.com>
Date: Wed, 22 Nov, 2023, 3:07 pm
Subject: How to Achieve Your Career Goals in 2024
To: <alice123@test.com>


\\r\\n\\r\\n \\r\\n \\r\\n \\r\\n\\r\\n \\r\\n \\r\\n \\r\\n \\r\\n \\r\\n
\\r\\n
 
\\r\\n \\r\\n \\r\\n\",\"charset\":\"utf-8\",\"content_transfer_encoding\":\"quoted-printable\",\"inner_text\":\"---------- Forwarded message ---------\\nFrom:\\nxyz Instructor: Alice \\n<\\nno-reply@e.example.com\\n>\\nDate: Wed, 22 Nov, 2023, 3:07 pm\\nSubject: How to Achieve Your Career Goals in 2024\\nTo: <\\nalice123@test.com\\n>\\nNew Educational Announcement\\nHi Alice, an announcement has been made from Alice, instructor of\\nLinux Administration: The Complete Linux Bootcamp for 2023\\n.\\n\",\"display_text\":\"---------- Forwarded message ---------\\nFrom: xyz Instructor: \\nDate: Wed, 22 Nov, 2023, 3:07pm\\nSubject: How to Achieve Your Career Goals in 2024\\nTo: \\nNew Educational Announcement\\nHi Alice, an announcement has been made from Alice, instructor of Linux Administration: The Complete Linux Bootcamp for 2023.\"},\"plain\":{\"raw\":\"---------- Forwarded message ---------\\r\\nFrom: xyz Instructor: Alice \\r\\nDate: Wed, 22 Nov, 2023, 3:07 pm\\r\\nSubject: How to Achieve Your Career Goals in 2024\\r\\nTo: \\r\\n\\r\\n\\r\\n\\r\\n\\r\\n* New Educational Announcement *\\r\\n\\r\\n\",\"charset\":\"utf-8\",\"content_transfer_encoding\":\"quoted-printable\"},\"links\":[{\"display_text\":\"Linux Administration: The Complete Linux Bootcamp for 2023\",\"href_url\":{\"url\":\"https://e2.example.com/ls/click?upn=6n\",\"domain\":{\"domain\":\"e2.example.com\",\"root_domain\":\"example.com\",\"sld\":\"example\",\"subdomain\":\"e2\",\"tld\":\"com\",\"valid\":true},\"path\":\"/ls/click\",\"query_params\":\"upn=Z\",\"scheme\":\"https\"}},{\"display_text\":\"How I Went From Zero Job Offers To Working At xyz In 3 Years\",\"href_url\":{\"url\":\"https://e2.example.com/ls/click?upn=6n\",\"domain\":{\"domain\":\"e2.example.com\",\"root_domain\":\"example.com\",\"sld\":\"example\",\"subdomain\":\"e2\",\"tld\":\"com\",\"valid\":true},\"path\":\"/ls/click\",\"query_params\":\"upn=6n\",\"scheme\":\"https\"}},{\"display_text\":\"How to beat Imposter Syndrome!\",\"href_url\":{\"url\":\"https://e2.example.com/ls/click?upn=6nwDRRN3APL2A\",\"domain\":{\"domain\":\"e2.example.com\",\"root_domain\":\"example.com\",\"sld\":\"example\",\"subdomain\":\"e2\",\"tld\":\"com\",\"valid\":true},\"path\":\"/ls/click\",\"query_params\":\"upn=6nwDRRN3APL2A\",\"scheme\":\"https\"}},{\"display_text\":\"full state of AI for programmers\",\"href_url\":{\"url\":\"https://e2.example.com/ls/click?upn=6nwDR\",\"domain\":{\"domain\":\"e2.example.com\",\"root_domain\":\"example.com\",\"sld\":\"example\",\"subdomain\":\"e2\",\"tld\":\"com\",\"valid\":true},\"path\":\"/ls/click\",\"query_params\":\"upn=6nwDR\",\"scheme\":\"https\"}},{\"display_text\":\"See Announcement\",\"href_url\":{\"url\":\"https://e2.example.com/ls/click?upn=ZF3sOyS2SxEPIoSZT6Aoc\",\"domain\":{\"domain\":\"e2.example.com\",\"root_domain\":\"example.com\",\"sld\":\"example\",\"subdomain\":\"e2\",\"tld\":\"com\",\"valid\":true},\"path\":\"/ls/click\",\"query_params\":\"upn=ZF3sOyS2SxEPIoSZT6Aoc\",\"scheme\":\"https\"}},{\"display_text\":\"unsubscribe\",\"href_url\":{\"url\":\"https://e2.example.com/ls/click?upn=ZF3sOyS2S\",\"domain\":{\"domain\":\"e2.example.com\",\"root_domain\":\"example.com\",\"sld\":\"example\",\"subdomain\":\"e2\",\"tld\":\"com\",\"valid\":true},\"path\":\"/ls/click\",\"query_params\":\"upn=ZF3sOyS2S\",\"scheme\":\"https\"}},{\"display_text\":\"report abuse\",\"href_url\":{\"url\":\"https://e2.example.com/ls/click?upn=ZF3s\",\"domain\":{\"domain\":\"e2.example.com\",\"root_domain\":\"example.com\",\"sld\":\"example\",\"subdomain\":\"e2\",\"tld\":\"com\",\"valid\":true},\"path\":\"/ls/click\",\"query_params\":\"upn=ZF3s\",\"scheme\":\"https\"}},{\"display_url\":{\"url\":\"e.example.com\",\"domain\":{\"domain\":\"e.example.com\",\"root_domain\":\"example.com\",\"sld\":\"example\",\"subdomain\":\"e\",\"tld\":\"com\",\"valid\":true},\"scheme\":\"http\"},\"href_url\":{\"url\":\"e.example.com\",\"domain\":{\"domain\":\"e.example.com\",\"root_domain\":\"example.com\",\"sld\":\"example\",\"subdomain\":\"e\",\"tld\":\"com\",\"valid\":true},\"scheme\":\"http\"}},{\"display_url\":{\"url\":\"test.com\",\"domain\":{\"domain\":\"test.com\",\"root_domain\":\"test.com\",\"sld\":\"test\",\"tld\":\"com\",\"valid\":true},\"scheme\":\"http\"},\"href_url\":{\"url\":\"test.com\",\"domain\":{\"domain\":\"test.com\",\"root_domain\":\"test.com\",\"sld\":\"test\",\"tld\":\"com\",\"valid\":true},\"scheme\":\"http\"}}],\"current_thread\":{\"text\":\"\"}},\"external\":{\"created_at\":\"2024-08-08T06:39:47Z\",\"message_id\":\"11\",\"route_type\":\"received\",\"spam\":false},\"headers\":{\"auth_summary\":{\"dmarc\":{\"pass\":true,\"received_hop\":0,\"details\":{\"version\":null,\"verdict\":\"pass\",\"action\":null,\"policy\":\"NONE\",\"sub_policy\":\"QUARANTINE\",\"disposition\":\"NONE\",\"from\":{\"domain\":\"test.com\",\"root_domain\":\"test.com\",\"sld\":\"test\",\"tld\":\"com\",\"valid\":true}}},\"spf\":{\"pass\":true,\"error\":false,\"received_hop\":0,\"details\":{\"verdict\":\"pass\",\"server\":{\"domain\":\"test.com\",\"root_domain\":\"test.com\",\"sld\":\"test\",\"tld\":\"com\",\"valid\":true},\"client_ip\":{\"ip\":\"81.2.69.192\"},\"designator\":\"domain of alice123@test.com\",\"description\":\"test.com: domain of alice123@test.com designates 81.2.69.192 as permitted sender\"}}},\"date\":\"2024-08-08T06:39:33Z\",\"date_original_offset\":\"5\",\"delivered_to\":{\"email\":\"john123@test.com\",\"local_part\":\"john123\",\"domain\":{\"domain\":\"test.com\",\"root_domain\":\"test.com\",\"sld\":\"test\",\"tld\":\"com\",\"valid\":true}},\"domains\":[{\"domain\":\"mail-sor-f41.test.com\",\"root_domain\":\"test.com\",\"sld\":\"test\",\"subdomain\":\"mail-sor-f41\",\"tld\":\"com\",\"valid\":true},{\"domain\":\"mx.test.com\",\"root_domain\":\"test.com\",\"sld\":\"test\",\"subdomain\":\"mx\",\"tld\":\"com\",\"valid\":true},{\"domain\":\"test.com\",\"root_domain\":\"test.com\",\"sld\":\"test\",\"tld\":\"com\",\"valid\":true}],\"hops\":[{\"index\":0,\"authentication_results\":{\"compauth\": { \"verdict\": \"pass\", \"reason\": \"reason_value\" },\"type\":\"standard\",\"dkim\":\"pass\",\"dkim_details\":[{\"type\":\"dkim\",\"instance\":\"@test.com\",\"selector\":\"20230601\",\"signature\":\"elrBA+fb\"}],\"dmarc\":\"pass\",\"dmarc_details\":{\"version\":null,\"verdict\":\"pass\",\"action\":null,\"policy\":\"NONE\",\"sub_policy\":\"QUARANTINE\",\"disposition\":\"NONE\",\"from\":{\"domain\":\"test.com\",\"root_domain\":\"test.com\",\"sld\":\"test\",\"tld\":\"com\",\"valid\":true}},\"spf\":\"pass\",\"spf_details\":{\"verdict\":\"pass\",\"server\":{\"domain\":\"test.com\",\"root_domain\":\"test.com\",\"sld\":\"test\",\"tld\":\"com\",\"valid\":true},\"client_ip\":{\"ip\":\"81.2.69.192\"},\"designator\":\"alice123@test.com\",\"description\":\"test.com: domain of alice123@test.com designates 81.2.69.192 as permitted sender\"},\"server\":{\"domain\":\"mx.test.com\",\"root_domain\":\"test.com\",\"sld\":\"test\",\"subdomain\":\"mx\",\"tld\":\"com\",\"valid\":true}},\"signature\":{\"type\":\"dkim\",\"version\":\"1\",\"algorithm\":\"rsa-sha256\",\"selector\":\"20230601\",\"signature\":\"elrBA+fbKpLqfjEOj7s4cAR9f8s8Tz0lHFsCjgE4t5dSXKpL6YZRZ44EPNZ1I1ROJx jTmQ/zz41IUs7K4tWqtuJnM98GSCsEvc31Lr3w99k/kjWm0Raei5dWc/4OqxZKmEiV3J r4Un6IlHY5CiUO\",\"body_hash\":\"taAa59+CAu/Cmad8+7VJw9KpMlCezg5RHDG7X7f91P4=\",\"domain\":\"test.com\",\"headers\":\"to:subject:message-id:date:from:in-reply-to:references:mime-version :from:to:cc:subject:date:message-id:reply-to\"},\"received_spf\":{\"verdict\":\"pass\",\"server\":{\"domain\":\"test.com\",\"root_domain\":\"test.com\",\"sld\":\"test\",\"tld\":\"com\",\"valid\":true},\"client_ip\":{\"ip\":\"81.2.69.192\"},\"designator\":\"domain of alice123@test.com\",\"description\":\"test.com: domain of alice123@test.com designates 81.2.69.192 as permitted sender\"},\"fields\":[{\"name\":\"Content-Type\",\"value\":\"multipart/alternative; boundary=\\\"000000000000e2ee4a061f264a8a\\\"\",\"position\":0},{\"name\":\"To\",\"value\":\"john123@test.com\",\"position\":1},{\"name\":\"Subject\",\"value\":\"Fwd: How to Achieve Your Career Goals in 2024\",\"position\":2},{\"name\":\"Message-ID\",\"value\":\"\",\"position\":3},{\"name\":\"Date\",\"value\":\"Thu, 8 Aug 2024 12:09:33 +0530\",\"position\":4},{\"name\":\"From\",\"value\":\"Alice \",\"position\":5},{\"name\":\"In-Reply-To\",\"value\":\"\",\"position\":6},{\"name\":\"References\",\"value\":\"\",\"position\":7},{\"name\":\"MIME-Version\",\"value\":\"1.0\",\"position\":8},{\"name\":\"X-Received\",\"value\":\"by 2002:a05:6512:12cc:b0:52b:faa1:7c74 with SMTP id 2adb3069b0e04-530e5d21173mr181491e87.5.1723099185866; Wed, 07 Aug 2024 23:39:45 -0700 (PDT)\",\"position\":9},{\"name\":\"X-Test-Smtp-Source\",\"value\":\"AGHT+IEOPD6/KQ1QGxfpqSsNfRVL3tyvz5l2ZkiY/swtxVoJf6PjrsQMELt982FDdWiYdCh6nHFEnpPzh50Rx+gpG+8=\",\"position\":10},{\"name\":\"X-Gm-Message-State\",\"value\":\"AOJu0YyLnr/Zjgr7uXbT4OWJKXOdcruE0DB5eYAxH2GMsUDqfM81HEem KMt/4qXMzEYtzJJ+2fkjA2zpg2e3zN9iqNuiUI86WAOTl07NfeqPMJCIDDz0Q0gfeiOwIK8rlL8 jammp+/G9xCHMayr+inDiqemweFRyvA==\",\"position\":11},{\"name\":\"X-Test-DKIM-Signature\",\"value\":\"v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723099186; x=1723703986; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=taAa59+CAu/Cmad8+7VJw9KpMlCezg5RHDG7X7f91P4=; b=VcRvW5nfz7WbzaIuyv5g5x/Z2U0qbrDR6qP\",\"position\":12},{\"name\":\"DKIM-Signature\",\"value\":\"v=1; a=rsa-sha256; c=relaxed/relaxed; d=test.com; s=20230601; t=1723099186; x=1723703986; dara=test.com; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :from:to:cc:subject:date:message-id:reply-to; bh=taAa59+CAu/Cmad8+7VJw9KpMlCezg5RHDG7X7f91P4=\",\"position\":13},{\"name\":\"Authentication-Results\",\"value\":\"mx.test.com; dkim=pass header.i=@test.com header.s=20230601 header.b=elrBA+fb; spf=pass (test.com: domain of alice123@test.com designates 81.2.69.192 as permitted sender) smtp.mailfrom=alice123@test.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=test.com; dara=pass header.i=@test.com\",\"position\":14},{\"name\":\"Received-SPF\",\"value\":\"pass (test.com: domain of alice123@test.com designates 81.2.69.192 as permitted sender) client-ip=81.2.69.192;\",\"position\":15},{\"name\":\"Received\",\"value\":\"from mail-sor-f41.test.com (mail-sor-f41.test.com. [81.2.69.192]) by mx.test.com with SMTPS id 2adb3069b0e04-530de79d133sor508979e87.12.2024.08.07.23.39.46 for (Test Transport Security); Wed, 07 Aug 2024 23:39:47 -0700 (PDT)\",\"position\":16}],\"received\":{\"source\":{\"raw\":\"mail-sor-f41.test.com (mail-sor-f41.test.com. [81.2.69.192])\"},\"server\":{\"raw\":\"mx.test.com\"},\"mailbox\":{\"raw\":\"\"},\"protocol\":{\"raw\":\"SMTPS\"},\"id\":{\"raw\":\"2adb3069b1e04-530de79d133sor508979e87.12.2024.08.07.23.39.46\"},\"additional\":{\"raw\":\"(Test Transport Security)\"},\"time\":\"2024-08-08T06:39:47Z\",\"zone_offset\":\"-7\"}},{\"index\":1,\"authentication_results\":{\"type\":\"arc\",\"instance\":\"1\",\"dkim\":\"pass\",\"dkim_details\":[{\"type\":\"dkim\",\"instance\":\"@test.com\",\"selector\":\"20230601\",\"signature\":\"elrBA+fb\"}],\"dmarc\":\"pass\",\"dmarc_details\":{\"version\":null,\"verdict\":\"pass\",\"action\":null,\"policy\":\"NONE\",\"sub_policy\":\"QUARANTINE\",\"disposition\":\"NONE\",\"from\":{\"domain\":\"test.com\",\"root_domain\":\"test.com\",\"sld\":\"test\",\"tld\":\"com\",\"valid\":true}},\"spf\":\"pass\",\"spf_details\":{\"verdict\":\"pass\",\"server\":{\"domain\":\"test.com\",\"root_domain\":\"test.com\",\"sld\":\"test\",\"tld\":\"com\",\"valid\":true},\"client_ip\":{\"ip\":\"81.2.69.192\"},\"designator\":\"alice123@test.com\",\"description\":\"test.com: domain of alice123@test.com designates 81.2.69.192 as permitted sender\"},\"server\":{\"domain\":\"mx.test.com\",\"root_domain\":\"test.com\",\"sld\":\"test\",\"subdomain\":\"mx\",\"tld\":\"com\",\"valid\":true}},\"signature\":{\"type\":\"arc-message\",\"instance\":\"1\",\"algorithm\":\"rsa-sha256\",\"selector\":\"arc-20160816\",\"signature\":\"nKqpy2hvLAXWHwdm39Mg1dL6lziVFqVY7ikY9FaP1w0pDHO6t0zbiMwcwSkS/Crz+ Y38+/FHiPhk65AocA0Yzw9P96RpK60iDaHfXpEBsxJIhJt9GN7\",\"body_hash\":\"taAa59+CAu/Cmad8+7VJw9KpMlCezg5RHDG7X7f91P4=\",\"domain\":\"test.com\",\"headers\":\"to:subject:message-id:date:from:in-reply-to:references:mime-version :dkim-signature\"},\"fields\":[{\"name\":\"Return-Path\",\"value\":\"\",\"position\":17},{\"name\":\"ARC-Authentication-Results\",\"value\":\"i=1; mx.test.com; dkim=pass header.i=@test.com header.s=20230601 header.b=elrBA+fb; spf=pass (test.com: domain of alice123@test.com designates 81.2.69.192 as permitted sender) smtp.mailfrom=alice123@test.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=test.com; dara=pass header.i=@test.com\",\"position\":18},{\"name\":\"ARC-Message-Signature\",\"value\":\"i=1; a=rsa-sha256; c=relaxed/relaxed; d=test.com; s=arc-20160816; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :dkim-signature; bh=taAa59+CAu/Cmad8+7VJw9KpMlCezg5RHDG7X7f91P4=; fh=rEZ8N7lZwF+f2DJz7PZfSiLZqwmiZLvrdguxWR5M0Mw=; b=nKqpy2hvLAXWHw39Mg1dLY6lziVFqVY7ikY9FaP1\",\"position\":19},{\"name\":\"ARC-Seal\",\"value\":\"i=1; a=rsa-sha256; t=1723099187; cv=none; d=test.com; s=arc-20160816; b=s4+J1/60S5sNdJ0Fd56rNghLRYU+m7QHad7No6E0iBi+7WGCuOOd2w07CSfEx++0jx Y0lBuDGDzNrGGHVpfi3ODGvx/aoU2vg8/siNaHAnIR4ADSbV+sr67vFiIEPqYNmQyihC lhOm0gnxDVD7ozZ\",\"position\":20},{\"name\":\"X-Received\",\"value\":\"by 2002:a05:6512:1242:b0:52f:c398:8780 with SMTP id 2adb3069b0e04-530e5d70e42mr184435e87.18.1723099187103; Wed, 07 Aug 2024 23:39:47 -0700 (PDT)\",\"position\":21},{\"name\":\"Received\",\"value\":\"by 2002:a05:6850:988a:b0:5bb:ddaf:ae20 with SMTP id li10csp662340nnb; Wed, 7 Aug 2024 23:39:47 -0700 (PDT)\",\"position\":22}],\"received\":{\"server\":{\"raw\":\"2002:a05:6850:988a:b0:5bb:ddaf:ae20\"},\"protocol\":{\"raw\":\"SMTP\"},\"id\":{\"raw\":\"li10csp662340nnb\"},\"time\":\"2024-08-08T06:39:47Z\",\"zone_offset\":\"-7\"}},{\"index\":2,\"fields\":[{\"name\":\"Delivered-To\",\"value\":\"john123@test.com\",\"position\":23}]}],\"ips\":[{\"ip\":\"81.2.69.192\"}],\"message_id\":\"\",\"references\":[\"hpfhx9h8QtSRnWCE_AzviQ@geopod-ismtpd-56\"],\"in_reply_to\":\"\",\"return_path\":{\"email\":\"alice123@test.com\",\"local_part\":\"alice123\",\"domain\":{\"domain\":\"test.com\",\"root_domain\":\"test.com\",\"sld\":\"test\",\"tld\":\"com\",\"valid\":true}}},\"type\":{\"inbound\":true},\"mailbox\":{\"email\":{\"email\":\"john123@test.com\",\"local_part\":\"john123\",\"domain\":{\"domain\":\"test.com\",\"root_domain\":\"test.com\",\"sld\":\"test\",\"tld\":\"com\",\"valid\":true}}},\"recipients\":{\"to\":[{\"email\":{\"email\":\"john123@test.com\",\"local_part\":\"john123\",\"domain\":{\"domain\":\"test.com\",\"root_domain\":\"test.com\",\"sld\":\"test\",\"tld\":\"com\",\"valid\":true}}}]},\"sender\":{\"display_name\":\"Alice\",\"email\":{\"email\":\"alice123@test.com\",\"local_part\":\"alice123\",\"domain\":{\"domain\":\"test.com\",\"root_domain\":\"test.com\",\"sld\":\"test\",\"tld\":\"com\",\"valid\":true}}},\"subject\":{\"subject\":\"Fwd: How to Achieve Your Career Goals in 2024\"},\"_meta\":{\"id\":\"019130be-779c-7641-87c7-284e8dcb10e5\",\"canonical_id\":\"dc767a9c58a14ea5560b5786c644d5cb9ef7cadb11f680231ec09e8fdfeb4d53\",\"created_at\":\"2024-08-08T06:47:54.460399216Z\",\"effective_at\":\"2024-08-08T06:39:47Z\"}}", + "type": [ + "info" + ] + }, + "observer": { + "product": "Sublime Security", + "vendor": "Sublime Security" + }, + "related": { + "hosts": [ + "e.example.com", + "test.com", + "example.com", + "e2.example.com", + "mail-sor-f41.test.com", + "mx.test.com" + ], + "ip": [ + "81.2.69.192" + ], + "user": [ + "john123@test.com", + "alice123@test.com" + ] + }, + "source": { + "domain": "test.com", + "top_level_domain": "com" + }, + "sublime_security": { + "email_message": { + "body": { + "html": { + "charset": "utf-8", + "content_transfer_encoding": "quoted-printable", + "display_text": "---------- Forwarded message ---------\nFrom: xyz Instructor: \nDate: Wed, 22 Nov, 2023, 3:07pm\nSubject: How to Achieve Your Career Goals in 2024\nTo: \nNew Educational Announcement\nHi Alice, an announcement has been made from Alice, instructor of Linux Administration: The Complete Linux Bootcamp for 2023.", + "inner_text": "---------- Forwarded message ---------\nFrom:\nxyz Instructor: Alice \n<\nno-reply@e.example.com\n>\nDate: Wed, 22 Nov, 2023, 3:07 pm\nSubject: How to Achieve Your Career Goals in 2024\nTo: <\nalice123@test.com\n>\nNew Educational Announcement\nHi Alice, an announcement has been made from Alice, instructor of\nLinux Administration: The Complete Linux Bootcamp for 2023\n.\n", + "raw": "


---------- Forwarded message ---------
From: xyz Instructor: Alice <no-reply@e.example.com>
Date: Wed, 22 Nov, 2023, 3:07 pm
Subject: How to Achieve Your Career Goals in 2024
To: <alice123@test.com>


\r\n\r\n \r\n \r\n \r\n\r\n \r\n \r\n \r\n \r\n \r\n
\r\n
 
\r\n \r\n \r\n" + }, + "links": [ + { + "display_text": "Linux Administration: The Complete Linux Bootcamp for 2023", + "href_url": { + "domain": { + "domain": "e2.example.com", + "root_domain": "example.com", + "sld": "example", + "subdomain": "e2", + "tld": "com", + "valid": true + }, + "path": "/ls/click", + "query_params": "upn=Z", + "scheme": "https", + "url": "https://e2.example.com/ls/click?upn=6n" + } + }, + { + "display_text": "How I Went From Zero Job Offers To Working At xyz In 3 Years", + "href_url": { + "domain": { + "domain": "e2.example.com", + "root_domain": "example.com", + "sld": "example", + "subdomain": "e2", + "tld": "com", + "valid": true + }, + "path": "/ls/click", + "query_params": "upn=6n", + "scheme": "https", + "url": "https://e2.example.com/ls/click?upn=6n" + } + }, + { + "display_text": "How to beat Imposter Syndrome!", + "href_url": { + "domain": { + "domain": "e2.example.com", + "root_domain": "example.com", + "sld": "example", + "subdomain": "e2", + "tld": "com", + "valid": true + }, + "path": "/ls/click", + "query_params": "upn=6nwDRRN3APL2A", + "scheme": "https", + "url": "https://e2.example.com/ls/click?upn=6nwDRRN3APL2A" + } + }, + { + "display_text": "full state of AI for programmers", + "href_url": { + "domain": { + "domain": "e2.example.com", + "root_domain": "example.com", + "sld": "example", + "subdomain": "e2", + "tld": "com", + "valid": true + }, + "path": "/ls/click", + "query_params": "upn=6nwDR", + "scheme": "https", + "url": "https://e2.example.com/ls/click?upn=6nwDR" + } + }, + { + "display_text": "See Announcement", + "href_url": { + "domain": { + "domain": "e2.example.com", + "root_domain": "example.com", + "sld": "example", + "subdomain": "e2", + "tld": "com", + "valid": true + }, + "path": "/ls/click", + "query_params": "upn=ZF3sOyS2SxEPIoSZT6Aoc", + "scheme": "https", + "url": "https://e2.example.com/ls/click?upn=ZF3sOyS2SxEPIoSZT6Aoc" + } + }, + { + "display_text": "unsubscribe", + "href_url": { + "domain": { + "domain": "e2.example.com", + "root_domain": "example.com", + "sld": "example", + "subdomain": "e2", + "tld": "com", + "valid": true + }, + "path": "/ls/click", + "query_params": "upn=ZF3sOyS2S", + "scheme": "https", + "url": "https://e2.example.com/ls/click?upn=ZF3sOyS2S" + } + }, + { + "display_text": "report abuse", + "href_url": { + "domain": { + "domain": "e2.example.com", + "root_domain": "example.com", + "sld": "example", + "subdomain": "e2", + "tld": "com", + "valid": true + }, + "path": "/ls/click", + "query_params": "upn=ZF3s", + "scheme": "https", + "url": "https://e2.example.com/ls/click?upn=ZF3s" + } + }, + { + "display_url": { + "domain": { + "domain": "e.example.com", + "root_domain": "example.com", + "sld": "example", + "subdomain": "e", + "tld": "com", + "valid": true + }, + "scheme": "http", + "url": "e.example.com" + }, + "href_url": { + "domain": { + "domain": "e.example.com", + "root_domain": "example.com", + "sld": "example", + "subdomain": "e", + "tld": "com", + "valid": true + }, + "scheme": "http", + "url": "e.example.com" + } + }, + { + "display_url": { + "domain": { + "domain": "test.com", + "root_domain": "test.com", + "sld": "test", + "tld": "com", + "valid": true + }, + "scheme": "http", + "url": "test.com" + }, + "href_url": { + "domain": { + "domain": "test.com", + "root_domain": "test.com", + "sld": "test", + "tld": "com", + "valid": true + }, + "scheme": "http", + "url": "test.com" + } + } + ], + "plain": { + "charset": "utf-8", + "content_transfer_encoding": "quoted-printable", + "raw": "---------- Forwarded message ---------\r\nFrom: xyz Instructor: Alice \r\nDate: Wed, 22 Nov, 2023, 3:07 pm\r\nSubject: How to Achieve Your Career Goals in 2024\r\nTo: \r\n\r\n\r\n\r\n\r\n* New Educational Announcement *\r\n\r\n" + } + }, + "external": { + "created_at": "2024-08-08T06:39:47.000Z", + "message_id": "11", + "route_type": "received", + "spam": false + }, + "headers": { + "auth_summary": { + "dmarc": { + "details": { + "disposition": "NONE", + "from": { + "domain": "test.com", + "root_domain": "test.com", + "sld": "test", + "tld": "com", + "valid": true + }, + "policy": "NONE", + "sub_policy": "QUARANTINE", + "verdict": "pass" + }, + "pass": true, + "received_hop": 0 + }, + "spf": { + "details": { + "client_ip": { + "ip": "81.2.69.192" + }, + "description": "test.com: domain of alice123@test.com designates 81.2.69.192 as permitted sender", + "designator": "domain of alice123@test.com", + "server": { + "domain": "test.com", + "root_domain": "test.com", + "sld": "test", + "tld": "com", + "valid": true + }, + "verdict": "pass" + }, + "error": false, + "pass": true, + "received_hop": 0 + } + }, + "date": "2024-08-08T06:39:33.000Z", + "date_original_offset": "5", + "delivered_to": { + "domain": { + "domain": "test.com", + "root_domain": "test.com", + "sld": "test", + "tld": "com", + "valid": true + }, + "email": "john123@test.com", + "local_part": "john123" + }, + "domains": [ + { + "domain": "mail-sor-f41.test.com", + "root_domain": "test.com", + "sld": "test", + "subdomain": "mail-sor-f41", + "tld": "com", + "valid": true + }, + { + "domain": "mx.test.com", + "root_domain": "test.com", + "sld": "test", + "subdomain": "mx", + "tld": "com", + "valid": true + }, + { + "domain": "test.com", + "root_domain": "test.com", + "sld": "test", + "tld": "com", + "valid": true + } + ], + "hops": [ + { + "authentication_results": { + "compauth": { + "reason": "reason_value", + "verdict": "pass" + }, + "dkim": "pass", + "dkim_details": [ + { + "instance": "@test.com", + "selector": "20230601", + "signature": "elrBA+fb", + "type": "dkim" + } + ], + "dmarc": "pass", + "dmarc_details": { + "disposition": "NONE", + "from": { + "domain": "test.com", + "root_domain": "test.com", + "sld": "test", + "tld": "com", + "valid": true + }, + "policy": "NONE", + "sub_policy": "QUARANTINE", + "verdict": "pass" + }, + "server": { + "domain": "mx.test.com", + "root_domain": "test.com", + "sld": "test", + "subdomain": "mx", + "tld": "com", + "valid": true + }, + "spf": "pass", + "spf_details": { + "client_ip": { + "ip": "81.2.69.192" + }, + "description": "test.com: domain of alice123@test.com designates 81.2.69.192 as permitted sender", + "designator": "alice123@test.com", + "server": { + "domain": "test.com", + "root_domain": "test.com", + "sld": "test", + "tld": "com", + "valid": true + }, + "verdict": "pass" + }, + "type": "standard" + }, + "fields": [ + { + "content-type": "multipart/alternative; boundary=\"000000000000e2ee4a061f264a8a\"", + "name": "Content-Type", + "position": 0, + "value": "multipart/alternative; boundary=\"000000000000e2ee4a061f264a8a\"" + }, + { + "name": "To", + "position": 1, + "to": "john123@test.com", + "value": "john123@test.com" + }, + { + "name": "Subject", + "position": 2, + "subject": "Fwd: How to Achieve Your Career Goals in 2024", + "value": "Fwd: How to Achieve Your Career Goals in 2024" + }, + { + "message-id": "", + "name": "Message-ID", + "position": 3, + "value": "" + }, + { + "date": "Thu, 8 Aug 2024 12:09:33 +0530", + "name": "Date", + "position": 4, + "value": "Thu, 8 Aug 2024 12:09:33 +0530" + }, + { + "from": "Alice ", + "name": "From", + "position": 5, + "value": "Alice " + }, + { + "in-reply-to": "", + "name": "In-Reply-To", + "position": 6, + "value": "" + }, + { + "name": "References", + "position": 7, + "references": "", + "value": "" + }, + { + "mime-version": "1.0", + "name": "MIME-Version", + "position": 8, + "value": "1.0" + }, + { + "name": "X-Received", + "position": 9, + "value": "by 2002:a05:6512:12cc:b0:52b:faa1:7c74 with SMTP id 2adb3069b0e04-530e5d21173mr181491e87.5.1723099185866; Wed, 07 Aug 2024 23:39:45 -0700 (PDT)", + "x-received": "by 2002:a05:6512:12cc:b0:52b:faa1:7c74 with SMTP id 2adb3069b0e04-530e5d21173mr181491e87.5.1723099185866; Wed, 07 Aug 2024 23:39:45 -0700 (PDT)" + }, + { + "name": "X-Test-Smtp-Source", + "position": 10, + "value": "AGHT+IEOPD6/KQ1QGxfpqSsNfRVL3tyvz5l2ZkiY/swtxVoJf6PjrsQMELt982FDdWiYdCh6nHFEnpPzh50Rx+gpG+8=", + "x-test-smtp-source": "AGHT+IEOPD6/KQ1QGxfpqSsNfRVL3tyvz5l2ZkiY/swtxVoJf6PjrsQMELt982FDdWiYdCh6nHFEnpPzh50Rx+gpG+8=" + }, + { + "name": "X-Gm-Message-State", + "position": 11, + "value": "AOJu0YyLnr/Zjgr7uXbT4OWJKXOdcruE0DB5eYAxH2GMsUDqfM81HEem KMt/4qXMzEYtzJJ+2fkjA2zpg2e3zN9iqNuiUI86WAOTl07NfeqPMJCIDDz0Q0gfeiOwIK8rlL8 jammp+/G9xCHMayr+inDiqemweFRyvA==", + "x-gm-message-state": "AOJu0YyLnr/Zjgr7uXbT4OWJKXOdcruE0DB5eYAxH2GMsUDqfM81HEem KMt/4qXMzEYtzJJ+2fkjA2zpg2e3zN9iqNuiUI86WAOTl07NfeqPMJCIDDz0Q0gfeiOwIK8rlL8 jammp+/G9xCHMayr+inDiqemweFRyvA==" + }, + { + "name": "X-Test-DKIM-Signature", + "position": 12, + "value": "v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723099186; x=1723703986; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=taAa59+CAu/Cmad8+7VJw9KpMlCezg5RHDG7X7f91P4=; b=VcRvW5nfz7WbzaIuyv5g5x/Z2U0qbrDR6qP", + "x-test-dkim-signature": "v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723099186; x=1723703986; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=taAa59+CAu/Cmad8+7VJw9KpMlCezg5RHDG7X7f91P4=; b=VcRvW5nfz7WbzaIuyv5g5x/Z2U0qbrDR6qP" + }, + { + "dkim-signature": "v=1; a=rsa-sha256; c=relaxed/relaxed; d=test.com; s=20230601; t=1723099186; x=1723703986; dara=test.com; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :from:to:cc:subject:date:message-id:reply-to; bh=taAa59+CAu/Cmad8+7VJw9KpMlCezg5RHDG7X7f91P4=", + "name": "DKIM-Signature", + "position": 13, + "value": "v=1; a=rsa-sha256; c=relaxed/relaxed; d=test.com; s=20230601; t=1723099186; x=1723703986; dara=test.com; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :from:to:cc:subject:date:message-id:reply-to; bh=taAa59+CAu/Cmad8+7VJw9KpMlCezg5RHDG7X7f91P4=" + }, + { + "authentication-results": "mx.test.com; dkim=pass header.i=@test.com header.s=20230601 header.b=elrBA+fb; spf=pass (test.com: domain of alice123@test.com designates 81.2.69.192 as permitted sender) smtp.mailfrom=alice123@test.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=test.com; dara=pass header.i=@test.com", + "name": "Authentication-Results", + "position": 14, + "value": "mx.test.com; dkim=pass header.i=@test.com header.s=20230601 header.b=elrBA+fb; spf=pass (test.com: domain of alice123@test.com designates 81.2.69.192 as permitted sender) smtp.mailfrom=alice123@test.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=test.com; dara=pass header.i=@test.com" + }, + { + "name": "Received-SPF", + "position": 15, + "received-spf": "pass (test.com: domain of alice123@test.com designates 81.2.69.192 as permitted sender) client-ip=81.2.69.192;", + "value": "pass (test.com: domain of alice123@test.com designates 81.2.69.192 as permitted sender) client-ip=81.2.69.192;" + }, + { + "name": "Received", + "position": 16, + "received": "from mail-sor-f41.test.com (mail-sor-f41.test.com. [81.2.69.192]) by mx.test.com with SMTPS id 2adb3069b0e04-530de79d133sor508979e87.12.2024.08.07.23.39.46 for (Test Transport Security); Wed, 07 Aug 2024 23:39:47 -0700 (PDT)", + "value": "from mail-sor-f41.test.com (mail-sor-f41.test.com. [81.2.69.192]) by mx.test.com with SMTPS id 2adb3069b0e04-530de79d133sor508979e87.12.2024.08.07.23.39.46 for (Test Transport Security); Wed, 07 Aug 2024 23:39:47 -0700 (PDT)" + } + ], + "index": 0, + "received": { + "additional": { + "raw": "(Test Transport Security)" + }, + "id": { + "raw": "2adb3069b1e04-530de79d133sor508979e87.12.2024.08.07.23.39.46" + }, + "mailbox": { + "raw": "" + }, + "protocol": { + "raw": "SMTPS" + }, + "server": { + "raw": "mx.test.com" + }, + "source": { + "raw": "mail-sor-f41.test.com (mail-sor-f41.test.com. [81.2.69.192])" + }, + "time": "2024-08-08T06:39:47.000Z", + "zone_offset": "-7" + }, + "received_spf": { + "client_ip": { + "ip": "81.2.69.192" + }, + "description": "test.com: domain of alice123@test.com designates 81.2.69.192 as permitted sender", + "designator": "domain of alice123@test.com", + "server": { + "domain": "test.com", + "root_domain": "test.com", + "sld": "test", + "tld": "com", + "valid": true + }, + "verdict": "pass" + }, + "signature": { + "algorithm": "rsa-sha256", + "body_hash": "taAa59+CAu/Cmad8+7VJw9KpMlCezg5RHDG7X7f91P4=", + "domain": "test.com", + "headers": "to:subject:message-id:date:from:in-reply-to:references:mime-version :from:to:cc:subject:date:message-id:reply-to", + "selector": "20230601", + "signature": "elrBA+fbKpLqfjEOj7s4cAR9f8s8Tz0lHFsCjgE4t5dSXKpL6YZRZ44EPNZ1I1ROJx jTmQ/zz41IUs7K4tWqtuJnM98GSCsEvc31Lr3w99k/kjWm0Raei5dWc/4OqxZKmEiV3J r4Un6IlHY5CiUO", + "type": "dkim", + "version": "1" + } + }, + { + "authentication_results": { + "dkim": "pass", + "dkim_details": [ + { + "instance": "@test.com", + "selector": "20230601", + "signature": "elrBA+fb", + "type": "dkim" + } + ], + "dmarc": "pass", + "dmarc_details": { + "disposition": "NONE", + "from": { + "domain": "test.com", + "root_domain": "test.com", + "sld": "test", + "tld": "com", + "valid": true + }, + "policy": "NONE", + "sub_policy": "QUARANTINE", + "verdict": "pass" + }, + "instance": "1", + "server": { + "domain": "mx.test.com", + "root_domain": "test.com", + "sld": "test", + "subdomain": "mx", + "tld": "com", + "valid": true + }, + "spf": "pass", + "spf_details": { + "client_ip": { + "ip": "81.2.69.192" + }, + "description": "test.com: domain of alice123@test.com designates 81.2.69.192 as permitted sender", + "designator": "alice123@test.com", + "server": { + "domain": "test.com", + "root_domain": "test.com", + "sld": "test", + "tld": "com", + "valid": true + }, + "verdict": "pass" + }, + "type": "arc" + }, + "fields": [ + { + "name": "Return-Path", + "position": 17, + "return-path": "", + "value": "" + }, + { + "arc-authentication-results": "i=1; mx.test.com; dkim=pass header.i=@test.com header.s=20230601 header.b=elrBA+fb; spf=pass (test.com: domain of alice123@test.com designates 81.2.69.192 as permitted sender) smtp.mailfrom=alice123@test.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=test.com; dara=pass header.i=@test.com", + "name": "ARC-Authentication-Results", + "position": 18, + "value": "i=1; mx.test.com; dkim=pass header.i=@test.com header.s=20230601 header.b=elrBA+fb; spf=pass (test.com: domain of alice123@test.com designates 81.2.69.192 as permitted sender) smtp.mailfrom=alice123@test.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=test.com; dara=pass header.i=@test.com" + }, + { + "arc-message-signature": "i=1; a=rsa-sha256; c=relaxed/relaxed; d=test.com; s=arc-20160816; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :dkim-signature; bh=taAa59+CAu/Cmad8+7VJw9KpMlCezg5RHDG7X7f91P4=; fh=rEZ8N7lZwF+f2DJz7PZfSiLZqwmiZLvrdguxWR5M0Mw=; b=nKqpy2hvLAXWHw39Mg1dLY6lziVFqVY7ikY9FaP1", + "name": "ARC-Message-Signature", + "position": 19, + "value": "i=1; a=rsa-sha256; c=relaxed/relaxed; d=test.com; s=arc-20160816; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :dkim-signature; bh=taAa59+CAu/Cmad8+7VJw9KpMlCezg5RHDG7X7f91P4=; fh=rEZ8N7lZwF+f2DJz7PZfSiLZqwmiZLvrdguxWR5M0Mw=; b=nKqpy2hvLAXWHw39Mg1dLY6lziVFqVY7ikY9FaP1" + }, + { + "arc-seal": "i=1; a=rsa-sha256; t=1723099187; cv=none; d=test.com; s=arc-20160816; b=s4+J1/60S5sNdJ0Fd56rNghLRYU+m7QHad7No6E0iBi+7WGCuOOd2w07CSfEx++0jx Y0lBuDGDzNrGGHVpfi3ODGvx/aoU2vg8/siNaHAnIR4ADSbV+sr67vFiIEPqYNmQyihC lhOm0gnxDVD7ozZ", + "name": "ARC-Seal", + "position": 20, + "value": "i=1; a=rsa-sha256; t=1723099187; cv=none; d=test.com; s=arc-20160816; b=s4+J1/60S5sNdJ0Fd56rNghLRYU+m7QHad7No6E0iBi+7WGCuOOd2w07CSfEx++0jx Y0lBuDGDzNrGGHVpfi3ODGvx/aoU2vg8/siNaHAnIR4ADSbV+sr67vFiIEPqYNmQyihC lhOm0gnxDVD7ozZ" + }, + { + "name": "X-Received", + "position": 21, + "value": "by 2002:a05:6512:1242:b0:52f:c398:8780 with SMTP id 2adb3069b0e04-530e5d70e42mr184435e87.18.1723099187103; Wed, 07 Aug 2024 23:39:47 -0700 (PDT)", + "x-received": "by 2002:a05:6512:1242:b0:52f:c398:8780 with SMTP id 2adb3069b0e04-530e5d70e42mr184435e87.18.1723099187103; Wed, 07 Aug 2024 23:39:47 -0700 (PDT)" + }, + { + "name": "Received", + "position": 22, + "received": "by 2002:a05:6850:988a:b0:5bb:ddaf:ae20 with SMTP id li10csp662340nnb; Wed, 7 Aug 2024 23:39:47 -0700 (PDT)", + "value": "by 2002:a05:6850:988a:b0:5bb:ddaf:ae20 with SMTP id li10csp662340nnb; Wed, 7 Aug 2024 23:39:47 -0700 (PDT)" + } + ], + "index": 1, + "received": { + "id": { + "raw": "li10csp662340nnb" + }, + "protocol": { + "raw": "SMTP" + }, + "server": { + "raw": "2002:a05:6850:988a:b0:5bb:ddaf:ae20" + }, + "time": "2024-08-08T06:39:47.000Z", + "zone_offset": "-7" + }, + "signature": { + "algorithm": "rsa-sha256", + "body_hash": "taAa59+CAu/Cmad8+7VJw9KpMlCezg5RHDG7X7f91P4=", + "domain": "test.com", + "headers": "to:subject:message-id:date:from:in-reply-to:references:mime-version :dkim-signature", + "instance": "1", + "selector": "arc-20160816", + "signature": "nKqpy2hvLAXWHwdm39Mg1dL6lziVFqVY7ikY9FaP1w0pDHO6t0zbiMwcwSkS/Crz+ Y38+/FHiPhk65AocA0Yzw9P96RpK60iDaHfXpEBsxJIhJt9GN7", + "type": "arc-message" + } + }, + { + "fields": [ + { + "delivered-to": "john123@test.com", + "name": "Delivered-To", + "position": 23, + "value": "john123@test.com" + } + ], + "index": 2 + } + ], + "in_reply_to": "", + "ips": [ + { + "ip": "81.2.69.192" + } + ], + "message_id": "", + "references": [ + "hpfhx9h8QtSRnWCE_AzviQ@geopod-ismtpd-56" + ], + "return_path": { + "domain": { + "domain": "test.com", + "root_domain": "test.com", + "sld": "test", + "tld": "com", + "valid": true + }, + "email": "alice123@test.com", + "local_part": "alice123" + } + }, + "mailbox": { + "email": { + "domain": { + "domain": "test.com", + "root_domain": "test.com", + "sld": "test", + "tld": "com", + "valid": true + }, + "local_part": "john123", + "value": "john123@test.com" + } + }, + "meta": { + "canonical_id": "dc767a9c58a14ea5560b5786c644d5cb9ef7cadb11f680231ec09e8fdfeb4d53", + "created_at": "2024-08-08T06:47:54.460Z", + "effective_at": "2024-08-08T06:39:47.000Z", + "id": "019130be-779c-7641-87c7-284e8dcb10e5" + }, + "recipients": { + "to": [ + { + "email": { + "domain": { + "domain": "test.com", + "root_domain": "test.com", + "sld": "test", + "tld": "com", + "valid": true + }, + "local_part": "john123", + "value": "john123@test.com" + } + } + ] + }, + "sender": { + "display_name": "Alice", + "email": { + "domain": { + "domain": "test.com", + "root_domain": "test.com", + "sld": "test", + "tld": "com", + "valid": true + }, + "local_part": "alice123", + "value": "alice123@test.com" + } + }, + "subject": { + "subject": "Fwd: How to Achieve Your Career Goals in 2024" + }, + "type": { + "inbound": true + } + } + }, + "tags": [ + "preserve_duplicate_custom_fields" + ], + "url": [ + { + "domain": "e2.example.com", + "full": "https://e2.example.com/ls/click?upn=6n", + "path": "/ls/click", + "query": "upn=Z", + "scheme": "https", + "subdomain": "e2", + "top_level_domain": "com" + }, + { + "domain": "e2.example.com", + "full": "https://e2.example.com/ls/click?upn=6n", + "path": "/ls/click", + "query": "upn=6n", + "scheme": "https", + "subdomain": "e2", + "top_level_domain": "com" + }, + { + "domain": "e2.example.com", + "full": "https://e2.example.com/ls/click?upn=6nwDRRN3APL2A", + "path": "/ls/click", + "query": "upn=6nwDRRN3APL2A", + "scheme": "https", + "subdomain": "e2", + "top_level_domain": "com" + }, + { + "domain": "e2.example.com", + "full": "https://e2.example.com/ls/click?upn=6nwDR", + "path": "/ls/click", + "query": "upn=6nwDR", + "scheme": "https", + "subdomain": "e2", + "top_level_domain": "com" + }, + { + "domain": "e2.example.com", + "full": "https://e2.example.com/ls/click?upn=ZF3sOyS2SxEPIoSZT6Aoc", + "path": "/ls/click", + "query": "upn=ZF3sOyS2SxEPIoSZT6Aoc", + "scheme": "https", + "subdomain": "e2", + "top_level_domain": "com" + }, + { + "domain": "e2.example.com", + "full": "https://e2.example.com/ls/click?upn=ZF3sOyS2S", + "path": "/ls/click", + "query": "upn=ZF3sOyS2S", + "scheme": "https", + "subdomain": "e2", + "top_level_domain": "com" + }, + { + "domain": "e2.example.com", + "full": "https://e2.example.com/ls/click?upn=ZF3s", + "path": "/ls/click", + "query": "upn=ZF3s", + "scheme": "https", + "subdomain": "e2", + "top_level_domain": "com" + }, + { + "domain": "e.example.com", + "full": "e.example.com", + "scheme": "http", + "subdomain": "e", + "top_level_domain": "com" + }, + { + "domain": "test.com", + "full": "test.com", + "scheme": "http", + "top_level_domain": "com" + } + ] + } + ] +} \ No newline at end of file diff --git a/packages/sublime_security/data_stream/email_message/_dev/test/system/test-aws-s3-config.yml b/packages/sublime_security/data_stream/email_message/_dev/test/system/test-aws-s3-config.yml new file mode 100644 index 00000000000..bb6b6f37112 --- /dev/null +++ b/packages/sublime_security/data_stream/email_message/_dev/test/system/test-aws-s3-config.yml @@ -0,0 +1,14 @@ +input: aws-s3 +wait_for_data_timeout: 20m +vars: + access_key_id: "{{AWS_ACCESS_KEY_ID}}" + secret_access_key: "{{AWS_SECRET_ACCESS_KEY}}" + session_token: "{{AWS_SESSION_TOKEN}}" +data_stream: + vars: + queue_url: "{{TF_OUTPUT_queue_url}}" + preserve_original_event: true + file_selectors: | + - regex: '^(.+?)\.log' + assert: + hit_count: 1 diff --git a/packages/sublime_security/data_stream/email_message/agent/stream/aws-s3.yml.hbs b/packages/sublime_security/data_stream/email_message/agent/stream/aws-s3.yml.hbs new file mode 100644 index 00000000000..8fbc7eee6de --- /dev/null +++ b/packages/sublime_security/data_stream/email_message/agent/stream/aws-s3.yml.hbs @@ -0,0 +1,94 @@ +{{#if collect_s3_logs}} + +{{#if bucket_arn}} +bucket_arn: {{bucket_arn}} +{{/if}} +{{#if number_of_workers}} +number_of_workers: {{number_of_workers}} +{{/if}} +{{#if interval}} +bucket_list_interval: {{interval}} +{{/if}} +{{#if bucket_list_prefix}} +bucket_list_prefix: {{bucket_list_prefix}} +{{/if}} + +{{else}} + +{{#if queue_url}} +queue_url: {{queue_url}} +{{/if}} +{{#if region}} +region: {{region}} +{{/if}} +{{#if visibility_timeout}} +visibility_timeout: {{visibility_timeout}} +{{/if}} +{{#if api_timeout}} +api_timeout: {{api_timeout}} +{{/if}} +{{#if max_number_of_messages}} +max_number_of_messages: {{max_number_of_messages}} +{{/if}} +{{#if file_selectors}} +file_selectors: +{{file_selectors}} +{{/if}} + +{{/if}} + +{{#if access_key_id}} +access_key_id: {{access_key_id}} +{{/if}} +{{#if secret_access_key}} +secret_access_key: {{secret_access_key}} +{{/if}} +{{#if session_token}} +session_token: {{session_token}} +{{/if}} +{{#if shared_credential_file}} +shared_credential_file: {{shared_credential_file}} +{{/if}} +{{#if credential_profile_name}} +credential_profile_name: {{credential_profile_name}} +{{/if}} +{{#if role_arn}} +role_arn: {{role_arn}} +{{/if}} +{{#if external_id}} +external_id: {{external_id}} +{{/if}} +{{#if default_region}} +default_region: {{default_region}} +{{/if}} +{{#if fips_enabled}} +fips_enabled: {{fips_enabled}} +{{/if}} +{{#if proxy_url}} +proxy_url: {{proxy_url}} +{{/if}} +{{#if ssl}} +ssl: {{ssl}} +{{/if}} +tags: +{{#if collect_s3_logs}} + - collect_s3_logs +{{else}} + - collect_sqs_logs +{{/if}} +{{#if preserve_original_event}} + - preserve_original_event +{{/if}} +{{#if preserve_duplicate_custom_fields}} + - preserve_duplicate_custom_fields +{{/if}} +{{#each tags as |tag|}} + - {{tag}} +{{/each}} +{{#contains "forwarded" tags}} +publisher_pipeline.disable_host: true +{{/contains}} +{{#if processors}} +processors: +{{processors}} +{{/if}} diff --git a/packages/sublime_security/data_stream/email_message/elasticsearch/ingest_pipeline/default.yml b/packages/sublime_security/data_stream/email_message/elasticsearch/ingest_pipeline/default.yml new file mode 100644 index 00000000000..83f3a843934 --- /dev/null +++ b/packages/sublime_security/data_stream/email_message/elasticsearch/ingest_pipeline/default.yml @@ -0,0 +1,2165 @@ +--- +description: Pipeline for processing email_message logs. +processors: + - set: + field: ecs.version + tag: set_ecs_version + value: 8.11.0 + - rename: + field: message + tag: rename_message_to_event_original + target_field: event.original + ignore_missing: true + description: Renames the original `message` field to `event.original` to store a copy of the original message. The `event.original` field is not touched if the document already has one; it may happen when Logstash sends the document. + if: ctx.event?.original == null + - remove: + field: message + tag: remove_message + ignore_missing: true + description: The `message` field is no longer required if the document has an `event.original` field. + if: ctx.event?.original != null + - json: + field: event.original + tag: json_event_original + target_field: json + on_failure: + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - set: + field: event.kind + tag: set_event_kind_to_event + value: event + - append: + field: event.category + tag: append_email_into_event_category + value: email + - append: + field: event.type + tag: append_info_into_event_type + value: info + - set: + field: observer.vendor + tag: set_observer_vendor + value: Sublime Security + - set: + field: observer.product + tag: set_observer_product + value: Sublime Security + - foreach: + field: json.attachments + if: ctx.json?.attachments instanceof List + processor: + rename: + field: _ingest._value.content_id + tag: rename_attachments_content_id + target_field: _ingest._value.content.id + ignore_missing: true + - foreach: + field: json.attachments + if: ctx.json?.attachments instanceof List + processor: + rename: + field: _ingest._value.content_transfer_encoding + tag: rename_attachments_content_transfer_encoding + target_field: _ingest._value.content.transfer_encoding + ignore_missing: true + - foreach: + field: json.attachments + if: ctx.json?.attachments instanceof List + processor: + rename: + field: _ingest._value.content_type + tag: rename_attachments_content_type + target_field: _ingest._value.content.type + ignore_missing: true + - foreach: + field: json.attachments + if: ctx.json?.attachments instanceof List + processor: + rename: + field: _ingest._value.file_name + tag: rename_attachments_file_name + target_field: _ingest._value.file.name + ignore_missing: true + - foreach: + field: json.attachments + if: ctx.json?.attachments instanceof List + processor: + rename: + field: _ingest._value.file_type + tag: rename_attachments_file_type + target_field: _ingest._value.file.type + ignore_missing: true + - foreach: + field: json.attachments + if: ctx.json?.attachments instanceof List + processor: + rename: + field: _ingest._value.file_extension + tag: rename_attachments_file_extension + target_field: _ingest._value.file.extension + ignore_missing: true + - foreach: + field: json.attachments + if: ctx.json?.attachments instanceof List + processor: + append: + field: related.hash + tag: append_attachments_md5_into_related_hash + value: '{{{_ingest._value.md5}}}' + allow_duplicates: false + - foreach: + field: json.attachments + if: ctx.json?.attachments instanceof List + processor: + append: + field: related.hash + tag: append_attachments_sha1_into_related_hash + value: '{{{_ingest._value.sha1}}}' + allow_duplicates: false + - foreach: + field: json.attachments + if: ctx.json?.attachments instanceof List + processor: + append: + field: related.hash + tag: append_attachments_sha256_into_related_hash + value: '{{{_ingest._value.sha256}}}' + allow_duplicates: false + - foreach: + field: json.attachments + if: ctx.json?.attachments instanceof List + processor: + convert: + field: _ingest._value.size + tag: convert_attachments_size_to_long + type: long + ignore_missing: true + on_failure: + - remove: + field: _ingest._value.size + ignore_missing: true + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - rename: + field: json.attachments + tag: rename_attachments + target_field: sublime_security.email_message.attachments + ignore_missing: true + - script: + description: Set email attachments field. + tag: script_to_set_email_attachments_field + lang: painless + if: ctx.sublime_security?.email_message?.attachments instanceof List + source: |- + def attachmentList = new ArrayList(); + for (attachment in ctx.sublime_security.email_message.attachments) { + def object = new HashMap(); + object.put('file', new HashMap()); + object.file.put('hash', new HashMap()); + object.file.put('mime_type', attachment.content.type); + object.file.put('extension', attachment.file.extension); + object.file.put('name', attachment.file.name); + object.file.put('size', attachment.size); + object.file.hash.put('md5', attachment.md5); + object.file.hash.put('sha1', attachment.sha1); + object.file.hash.put('sha256', attachment.sha256); + attachmentList.add(object); + } + ctx.put('email',new HashMap()); + ctx.email.attachments = attachmentList; + - script: + description: Extract extension name from email attachments. + tag: script_to_extract_extension_name + lang: painless + if: ctx.email?.attachments instanceof List + source: |- + for (attachment in ctx.email.attachments) { + if (attachment.file.extension != null && attachment.file.extension.startsWith('.')) { + String extension = attachment.file.extension.substring(1); + attachment.file.extension = extension; + } + } + - rename: + field: json.body.current_thread.text + tag: rename_body_current_thread_text + target_field: sublime_security.email_message.body.current_thread.text + ignore_missing: true + - rename: + field: json.body.html.charset + tag: rename_body_html_charset + target_field: sublime_security.email_message.body.html.charset + ignore_missing: true + - rename: + field: json.body.html.content_transfer_encoding + tag: rename_body_html_content_transfer_encoding + target_field: sublime_security.email_message.body.html.content_transfer_encoding + ignore_missing: true + - rename: + field: json.body.html.display_text + tag: rename_body_html_display_text + target_field: sublime_security.email_message.body.html.display_text + ignore_missing: true + - rename: + field: json.body.html.inner_text + tag: rename_body_html_inner_text + target_field: sublime_security.email_message.body.html.inner_text + ignore_missing: true + - rename: + field: json.body.html.raw + tag: rename_body_html_raw + target_field: sublime_security.email_message.body.html.raw + ignore_missing: true + - foreach: + field: json.body.ips + if: ctx.json?.body?.ips instanceof List + processor: + convert: + field: _ingest._value.ip + tag: convert_body_ips_ip_to_ip + type: ip + ignore_missing: true + on_failure: + - remove: + field: _ingest._value.ip + ignore_missing: true + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - foreach: + field: json.body.ips + if: ctx.json?.body?.ips instanceof List + processor: + append: + field: related.ip + tag: append_body_ips_ip_into_related_ip + value: '{{{_ingest._value.ip}}}' + allow_duplicates: false + - rename: + field: json.body.ips + tag: rename_body_ips + target_field: sublime_security.email_message.body.ips + ignore_missing: true + - foreach: + field: json.body.links + if: ctx.json?.body?.links instanceof List + processor: + append: + field: related.hosts + tag: append_body_links_display_url_domain_domain_into_related_hosts + value: '{{{_ingest._value.display_url.domain.domain}}}' + allow_duplicates: false + - foreach: + field: json.body.links + if: ctx.json?.body?.links instanceof List + processor: + convert: + field: _ingest._value.display_url.port + tag: convert_body_links_display_url_port_to_long + type: long + ignore_missing: true + on_failure: + - remove: + field: _ingest._value.display_url.port + ignore_missing: true + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - foreach: + field: json.body.links + if: ctx.json?.body?.links instanceof List + processor: + append: + field: related.hosts + tag: append_body_links_display_url_domain_root_domain_into_related_hosts + value: '{{{_ingest._value.display_url.domain.root_domain}}}' + allow_duplicates: false + - foreach: + field: json.body.links + if: ctx.json?.body?.links instanceof List + processor: + append: + field: related.user + tag: append_body_links_display_url_username_into_related_user + value: '{{{_ingest._value.display_url.username}}}' + allow_duplicates: false + - foreach: + field: json.body.links + if: ctx.json?.body?.links instanceof List + processor: + convert: + field: _ingest._value.display_url.domain.valid + tag: convert_body_links_display_url_domain_valid_to_boolean + type: boolean + ignore_missing: true + on_failure: + - remove: + field: _ingest._value.display_url.domain.valid + ignore_missing: true + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - foreach: + field: json.body.links + if: ctx.json?.body?.links instanceof List + processor: + append: + field: related.hosts + tag: append_body_links_href_url_domain_domain_into_related_hosts + value: '{{{_ingest._value.href_url.domain.domain}}}' + allow_duplicates: false + - foreach: + field: json.body.links + if: ctx.json?.body?.links instanceof List + processor: + append: + field: related.hosts + tag: append_body_links_href_url_domain_root_domain_into_related_hosts + value: '{{{_ingest._value.href_url.domain.root_domain}}}' + allow_duplicates: false + - foreach: + field: json.body.links + if: ctx.json?.body?.links instanceof List + processor: + convert: + field: _ingest._value.href_url.port + tag: convert_body_links_href_url_port_to_long + type: long + ignore_missing: true + on_failure: + - remove: + field: _ingest._value.href_url.port + ignore_missing: true + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - foreach: + field: json.body.links + if: ctx.json?.body?.links instanceof List + processor: + append: + field: related.user + tag: append_body_links_href_url_username_into_related.user + value: '{{{_ingest._value.href_url.username}}}' + allow_duplicates: false + - foreach: + field: json.body.links + if: ctx.json?.body?.links instanceof List + processor: + convert: + field: _ingest._value.href_url.domain.valid + tag: convert_body_links_href_url_domain_valid_to_boolean + type: boolean + ignore_missing: true + on_failure: + - remove: + field: _ingest._value.href_url.domain.valid + ignore_missing: true + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - foreach: + field: json.body.links + if: ctx.json?.body?.links instanceof List + processor: + convert: + field: _ingest._value.mismatched + tag: convert_body_links_mismatched_to_boolean + type: boolean + ignore_missing: true + on_failure: + - remove: + field: _ingest._value.mismatched + ignore_missing: true + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - script: + description: Set url field. + tag: script_to_set_url_field + lang: painless + if: ctx.json?.body?.links instanceof List + source: |- + def links = new ArrayList(); + for (link in ctx.json.body.links) { + def object = new HashMap(); + if(link?.href_url != null) { + object.put('domain', link.href_url.domain.domain); + object.put('subdomain', link.href_url.domain.subdomain); + object.put('top_level_domain', link.href_url.domain.tld); + object.put('fragment', link.href_url.fragment); + object.put('password', link.href_url.password); + object.put('path', link.href_url.path); + object.put('port', link.href_url.port); + object.put('query', link.href_url.query_params); + object.put('scheme', link.href_url.scheme); + object.put('full', link.href_url.url); + object.put('username', link.href_url.username); + links.add(object); + } + } + ctx.put('url',new HashMap()); + ctx.url = links; + - rename: + field: json.body.links + tag: rename_body_links + target_field: sublime_security.email_message.body.links + ignore_missing: true + - rename: + field: json.body.plain.charset + tag: rename_body_plain_charset + target_field: sublime_security.email_message.body.plain.charset + ignore_missing: true + - rename: + field: json.body.plain.content_transfer_encoding + tag: rename_body_plain_content_transfer_encoding + target_field: sublime_security.email_message.body.plain.content_transfer_encoding + ignore_missing: true + - rename: + field: json.body.plain.raw + tag: rename_body_plain_raw + target_field: sublime_security.email_message.body.plain.raw + ignore_missing: true + - rename: + field: json._errors + tag: rename_errors + target_field: sublime_security.email_message.errors + ignore_missing: true + - date: + field: json.external.created_at + tag: date_external_created_at + target_field: sublime_security.email_message.external.created_at + formats: + - ISO8601 + if: ctx.json?.external?.created_at != null && ctx.json.external.created_at != '' + on_failure: + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - set: + field: email.origination_timestamp + tag: set_email_origination_timestamp_from_email_message_external_created_at + copy_from: sublime_security.email_message.external.created_at + ignore_empty_value: true + - rename: + field: json.external.message_id + tag: rename_external_message_id + target_field: sublime_security.email_message.external.message_id + ignore_missing: true + - rename: + field: json.external.route_type + tag: rename_external_route_type + target_field: sublime_security.email_message.external.route_type + ignore_missing: true + - convert: + field: json.external.spam + tag: convert_external_spam_to_boolean + target_field: sublime_security.email_message.external.spam + type: boolean + ignore_missing: true + on_failure: + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - convert: + field: json.external.spam_folder + tag: convert_external_spam_folder_to_boolean + target_field: sublime_security.email_message.external.spam_folder + type: boolean + ignore_missing: true + on_failure: + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - rename: + field: json.external.thread_id + tag: rename_external_thread_id + target_field: sublime_security.email_message.external.thread_id + ignore_missing: true + - rename: + field: json.headers.auth_summary.dmarc.details.action + tag: rename_headers_auth_summary_dmarc_details_action + target_field: sublime_security.email_message.headers.auth_summary.dmarc.details.action + ignore_missing: true + - rename: + field: json.headers.auth_summary.dmarc.details.disposition + tag: rename_headers_auth_summary_dmarc_details_disposition + target_field: sublime_security.email_message.headers.auth_summary.dmarc.details.disposition + ignore_missing: true + - rename: + field: json.headers.auth_summary.dmarc.details.from.domain + tag: rename_headers_auth_summary_dmarc_details_from_domain + target_field: sublime_security.email_message.headers.auth_summary.dmarc.details.from.domain + ignore_missing: true + - append: + field: related.hosts + tag: append_headers_auth_summary_dmarc_details_from_domain_into_related_hosts + value: '{{{sublime_security.email_message.headers.auth_summary.dmarc.details.from.domain}}}' + allow_duplicates: false + - rename: + field: json.headers.auth_summary.dmarc.details.from.punycode + tag: rename_headers_auth_summary_dmarc_details_from_punycode + target_field: sublime_security.email_message.headers.auth_summary.dmarc.details.from.punycode + ignore_missing: true + - rename: + field: json.headers.auth_summary.dmarc.details.from.root_domain + tag: rename_headers_auth_summary_dmarc_details_from_root_domain + target_field: sublime_security.email_message.headers.auth_summary.dmarc.details.from.root_domain + ignore_missing: true + - append: + field: related.hosts + tag: append_headers_auth_summary_dmarc_details_from_root_domain_into_related_hosts + value: '{{{sublime_security.email_message.headers.auth_summary.dmarc.details.from.root_domain}}}' + allow_duplicates: false + - rename: + field: json.headers.auth_summary.dmarc.details.from.sld + tag: rename_headers_auth_summary_dmarc_details_from_sld + target_field: sublime_security.email_message.headers.auth_summary.dmarc.details.from.sld + ignore_missing: true + - rename: + field: json.headers.auth_summary.dmarc.details.from.subdomain + tag: rename_headers_auth_summary_dmarc_details_from_subdomain + target_field: sublime_security.email_message.headers.auth_summary.dmarc.details.from.subdomain + ignore_missing: true + - rename: + field: json.headers.auth_summary.dmarc.details.from.tld + tag: rename_headers_auth_summary_dmarc_details_from_tld + target_field: sublime_security.email_message.headers.auth_summary.dmarc.details.from.tld + ignore_missing: true + - convert: + field: json.headers.auth_summary.dmarc.details.from.valid + tag: convert_headers_auth_summary_dmarc_details_from_valid_to_boolean + target_field: sublime_security.email_message.headers.auth_summary.dmarc.details.from.valid + type: boolean + ignore_missing: true + on_failure: + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - rename: + field: json.headers.auth_summary.dmarc.details.policy + tag: rename_headers_auth_summary_dmarc_details_policy + target_field: sublime_security.email_message.headers.auth_summary.dmarc.details.policy + ignore_missing: true + - rename: + field: json.headers.auth_summary.dmarc.details.sub_policy + tag: rename_headers_auth_summary_dmarc_details_sub_policy + target_field: sublime_security.email_message.headers.auth_summary.dmarc.details.sub_policy + ignore_missing: true + - rename: + field: json.headers.auth_summary.dmarc.details.verdict + tag: rename_headers_auth_summary_dmarc_details_verdict + target_field: sublime_security.email_message.headers.auth_summary.dmarc.details.verdict + ignore_missing: true + - rename: + field: json.headers.auth_summary.dmarc.details.version + tag: rename_headers_auth_summary_dmarc_details_version + target_field: sublime_security.email_message.headers.auth_summary.dmarc.details.version + ignore_missing: true + - convert: + field: json.headers.auth_summary.dmarc.pass + tag: convert_headers_auth_summary_dmarc_pass_to_boolean + target_field: sublime_security.email_message.headers.auth_summary.dmarc.pass + type: boolean + ignore_missing: true + on_failure: + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - convert: + field: json.headers.auth_summary.dmarc.received_hop + tag: convert_headers_auth_summary_dmarc_received_hop_to_long + target_field: sublime_security.email_message.headers.auth_summary.dmarc.received_hop + type: long + ignore_missing: true + on_failure: + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - convert: + field: json.headers.auth_summary.spf.details.client_ip.ip + tag: convert_headers_auth_summary_spf_details_client_ip_ip_to_ip + target_field: sublime_security.email_message.headers.auth_summary.spf.details.client_ip.ip + type: ip + ignore_missing: true + if: ctx.json?.headers?.auth_summary?.spf?.details?.client_ip?.ip != null && ctx.json.headers.auth_summary.spf.details.client_ip.ip != '' + on_failure: + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - append: + field: related.ip + tag: append_headers_auth_summary_spf_details_client_ip_into_related_ip + value: '{{{sublime_security.email_message.headers.auth_summary.spf.details.client_ip.ip}}}' + allow_duplicates: false + - rename: + field: json.headers.auth_summary.spf.details.description + tag: rename_headers_auth_summary_spf_details_description + target_field: sublime_security.email_message.headers.auth_summary.spf.details.description + ignore_missing: true + - rename: + field: json.headers.auth_summary.spf.details.designator + tag: rename_headers_auth_summary_spf_details_designator + target_field: sublime_security.email_message.headers.auth_summary.spf.details.designator + ignore_missing: true + - rename: + field: json.headers.auth_summary.spf.details.helo.domain + tag: rename_headers_auth_summary_spf_details_helo_domain + target_field: sublime_security.email_message.headers.auth_summary.spf.details.helo.domain + ignore_missing: true + - append: + field: related.hosts + tag: append_headers_auth_summary_spf_details_helo_domain_into_related_host + value: '{{{sublime_security.email_message.headers.auth_summary.spf.details.helo.domain}}}' + allow_duplicates: false + - rename: + field: json.headers.auth_summary.spf.details.helo.punycode + tag: rename_headers_auth_summary_spf_details_helo_punycode + target_field: sublime_security.email_message.headers.auth_summary.spf.details.helo.punycode + ignore_missing: true + - rename: + field: json.headers.auth_summary.spf.details.helo.root_domain + tag: rename_headers_auth_summary_spf_details_helo_root_domain + target_field: sublime_security.email_message.headers.auth_summary.spf.details.helo.root_domain + ignore_missing: true + - append: + field: related.hosts + tag: append_headers_auth_summary_spf_details_helo_root_domain_into_related_host + value: '{{{sublime_security.email_message.headers.auth_summary.spf.details.helo.root_domain}}}' + allow_duplicates: false + - rename: + field: json.headers.auth_summary.spf.details.helo.sld + tag: rename_headers_auth_summary_spf_details_helo_sld + target_field: sublime_security.email_message.headers.auth_summary.spf.details.helo.sld + ignore_missing: true + - rename: + field: json.headers.auth_summary.spf.details.helo.subdomain + tag: rename_headers_auth_summary_spf_details_helo_subdomain + target_field: sublime_security.email_message.headers.auth_summary.spf.details.helo.subdomain + ignore_missing: true + - rename: + field: json.headers.auth_summary.spf.details.helo.tld + tag: rename_headers_auth_summary_spf_details_helo_tld + target_field: sublime_security.email_message.headers.auth_summary.spf.details.helo.tld + ignore_missing: true + - convert: + field: json.headers.auth_summary.spf.details.helo.valid + tag: convert_headers_auth_summary_spf_details_helo_valid_to_boolean + target_field: sublime_security.email_message.headers.auth_summary.spf.details.helo.valid + type: boolean + ignore_missing: true + on_failure: + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - rename: + field: json.headers.auth_summary.spf.details.server.domain + tag: rename_headers_auth_summary_spf_details_server_domain + target_field: sublime_security.email_message.headers.auth_summary.spf.details.server.domain + ignore_missing: true + - append: + field: related.hosts + tag: append_headers_auth_summary_spf_details_server_domain_into_related_host + value: '{{{sublime_security.email_message.headers.auth_summary.spf.details.server.domain}}}' + allow_duplicates: false + - rename: + field: json.headers.auth_summary.spf.details.server.punycode + tag: rename_headers_auth_summary_spf_details_server_punycode + target_field: sublime_security.email_message.headers.auth_summary.spf.details.server.punycode + ignore_missing: true + - rename: + field: json.headers.auth_summary.spf.details.server.root_domain + tag: rename_headers_auth_summary_spf_details_server_root_domain + target_field: sublime_security.email_message.headers.auth_summary.spf.details.server.root_domain + ignore_missing: true + - append: + field: related.hosts + tag: append_headers_auth_summary_spf_details_server_root_domain_into_related_host + value: '{{{sublime_security.email_message.headers.auth_summary.spf.details.server.root_domain}}}' + allow_duplicates: false + - rename: + field: json.headers.auth_summary.spf.details.server.sld + tag: rename_headers_auth_summary_spf_details_server_sld + target_field: sublime_security.email_message.headers.auth_summary.spf.details.server.sld + ignore_missing: true + - rename: + field: json.headers.auth_summary.spf.details.server.subdomain + tag: rename_headers_auth_summary_spf_details_server_subdomain + target_field: sublime_security.email_message.headers.auth_summary.spf.details.server.subdomain + ignore_missing: true + - rename: + field: json.headers.auth_summary.spf.details.server.tld + tag: rename_headers_auth_summary_spf_details_server_tld + target_field: sublime_security.email_message.headers.auth_summary.spf.details.server.tld + ignore_missing: true + - convert: + field: json.headers.auth_summary.spf.details.server.valid + tag: convert_headers_auth_summary_spf_details_server_valid_to_boolean + target_field: sublime_security.email_message.headers.auth_summary.spf.details.server.valid + type: boolean + ignore_missing: true + on_failure: + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - rename: + field: json.headers.auth_summary.spf.details.verdict + tag: rename_headers_auth_summary_spf_details_verdict + target_field: sublime_security.email_message.headers.auth_summary.spf.details.verdict + ignore_missing: true + - convert: + field: json.headers.auth_summary.spf.error + tag: convert_headers_auth_summary_spf_error_to_boolean + target_field: sublime_security.email_message.headers.auth_summary.spf.error + type: boolean + ignore_missing: true + on_failure: + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - convert: + field: json.headers.auth_summary.spf.pass + tag: convert_headers_auth_summary_spf_pass_to_boolean + target_field: sublime_security.email_message.headers.auth_summary.spf.pass + type: boolean + ignore_missing: true + on_failure: + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - convert: + field: json.headers.auth_summary.spf.received_hop + tag: convert_headers_auth_summary_spf_received_hop_to_long + target_field: sublime_security.email_message.headers.auth_summary.spf.received_hop + type: long + ignore_missing: true + on_failure: + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - date: + field: json.headers.date + tag: date_headers_date + target_field: sublime_security.email_message.headers.date + formats: + - ISO8601 + if: ctx.json?.headers?.date != null && ctx.json.headers.date != '' + on_failure: + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - rename: + field: json.headers.date_original_offset + tag: rename_headers_date_original_offset + target_field: sublime_security.email_message.headers.date_original_offset + ignore_missing: true + - rename: + field: json.headers.delivered_to.domain.domain + tag: rename_headers_delivered_to_domain_domain + target_field: sublime_security.email_message.headers.delivered_to.domain.domain + ignore_missing: true + - set: + field: destination.domain + tag: set_destination_domain_from_headers_delivered_to_domain_domain + copy_from: sublime_security.email_message.headers.delivered_to.domain.domain + ignore_empty_value: true + - rename: + field: json.headers.delivered_to.domain.punycode + tag: rename_headers_delivered_to_domain_punycode + target_field: sublime_security.email_message.headers.delivered_to.domain.punycode + ignore_missing: true + - rename: + field: json.headers.delivered_to.domain.root_domain + tag: rename_headers_delivered_to_domain_root_domain + target_field: sublime_security.email_message.headers.delivered_to.domain.root_domain + ignore_missing: true + - rename: + field: json.headers.delivered_to.domain.sld + tag: rename_headers_delivered_to_domain_sld + target_field: sublime_security.email_message.headers.delivered_to.domain.sld + ignore_missing: true + - rename: + field: json.headers.delivered_to.domain.subdomain + tag: rename_headers_delivered_to_domain_subdomain + target_field: sublime_security.email_message.headers.delivered_to.domain.subdomain + ignore_missing: true + - set: + field: destination.subdomain + tag: set_destination_subdomain_from_headers_delivered_to_domain_subdomain + copy_from: sublime_security.email_message.headers.delivered_to.domain.subdomain + ignore_empty_value: true + - rename: + field: json.headers.delivered_to.domain.tld + tag: rename_headers_delivered_to_domain_tld + target_field: sublime_security.email_message.headers.delivered_to.domain.tld + ignore_missing: true + - set: + field: destination.top_level_domain + tag: set_destination_top_level_domain_from_headers_delivered_to_domain_tld + copy_from: sublime_security.email_message.headers.delivered_to.domain.tld + ignore_empty_value: true + - convert: + field: json.headers.delivered_to.domain.valid + tag: convert_headers_delivered_to_domain_valid_to_boolean + target_field: sublime_security.email_message.headers.delivered_to.domain.valid + type: boolean + ignore_missing: true + on_failure: + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - rename: + field: json.headers.delivered_to.email + tag: rename_headers_delivered_to_email + target_field: sublime_security.email_message.headers.delivered_to.email + ignore_missing: true + - rename: + field: json.headers.delivered_to.local_part + tag: rename_headers_delivered_to_local_part + target_field: sublime_security.email_message.headers.delivered_to.local_part + ignore_missing: true + - foreach: + field: json.headers.domains + if: ctx.json?.headers?.domains instanceof List + processor: + append: + field: related.hosts + tag: append_headers_domains_domain_into_related_hosts + value: '{{{_ingest._value.domain}}}' + allow_duplicates: false + - foreach: + field: json.headers.domains + if: ctx.json?.headers?.domains instanceof List + processor: + append: + field: related.hosts + tag: append_headers_domains_root_domain_into_related_hosts + value: '{{{_ingest._value.root_domain}}}' + allow_duplicates: false + - foreach: + field: json.headers.domains + if: ctx.json?.headers?.domains instanceof List + processor: + convert: + field: _ingest._value.valid + tag: convert_headers_domains_valid_to_boolean + type: boolean + ignore_missing: true + on_failure: + - remove: + field: _ingest._value.valid + ignore_missing: true + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - rename: + field: json.headers.domains + tag: rename_headers_domains + target_field: sublime_security.email_message.headers.domains + ignore_missing: true + - foreach: + field: json.headers.hops + if: ctx.json?.headers?.hops instanceof List + processor: + convert: + field: _ingest._value.authentication_results.dmarc_details.from.valid + tag: convert_headers_hops_authentication_results_dmarc_details_from_valid_to_boolean + type: boolean + ignore_missing: true + on_failure: + - remove: + field: _ingest._value.authentication_results.dmarc_details.from.valid + ignore_missing: true + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - foreach: + field: json.headers.hops + if: ctx.json?.headers?.hops instanceof List + processor: + convert: + field: _ingest._value.authentication_results.server.valid + tag: convert_headers_hops_authentication_results_server_valid_to_boolean + type: boolean + ignore_missing: true + on_failure: + - remove: + field: _ingest._value.authentication_results.server.valid + ignore_missing: true + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - foreach: + field: json.headers.hops + if: ctx.json?.headers?.hops instanceof List + processor: + foreach: + field: _ingest._value.fields + ignore_failure: true + processor: + convert: + field: _ingest._value.position + tag: convert_headers_hops_fields_position_to_long + type: long + ignore_missing: true + on_failure: + - remove: + field: _ingest._value.position + ignore_missing: true + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - script: + lang: painless + description: add mapping of name and value in fields array. + tag: script to add mapping of name and value in fields array + if: ctx.json?.headers?.hops instanceof List + source: | + def hops = ctx.json.headers.hops; + for (int i = 0; i < hops.size(); i++) { + def hop = hops[i]; + if(hop.fields instanceof List) { + for (def field : hop.fields) { + def lowercaseName = field.name.toLowerCase(); + field[lowercaseName] = field.value; + } + } + } + - foreach: + field: json.headers.hops + if: ctx.json?.headers?.hops instanceof List + processor: + append: + field: related.hash + tag: append_headers_hops_authentication_results_dkim_details_body_hash_into_related_hash + value: '{{{_ingest._value.authentication_results.dkim_details.body_hash}}}' + allow_duplicates: false + - foreach: + field: json.headers.hops + if: ctx.json?.headers?.hops instanceof List + processor: + append: + field: related.hosts + tag: append_headers_hops_authentication_results_dmarc_details_from_domain_into_related_hosts + value: '{{{_ingest._value.authentication_results.dmarc_details.from.domain}}}' + allow_duplicates: false + - foreach: + field: json.headers.hops + if: ctx.json?.headers?.hops instanceof List + processor: + append: + field: related.hosts + tag: append_headers_hops_authentication_results_dmarc_details_from_root_domain_into_related_hosts + value: '{{{_ingest._value.authentication_results.dmarc_details.from.root_domain}}}' + allow_duplicates: false + - foreach: + field: json.headers.hops + if: ctx.json?.headers?.hops instanceof List + processor: + append: + field: related.hosts + tag: append_headers_hops_authentication_results_server_domain_into_related_hosts + value: '{{{_ingest._value.authentication_results.server.domain}}}' + allow_duplicates: false + - foreach: + field: json.headers.hops + if: ctx.json?.headers?.hops instanceof List + processor: + append: + field: related.hosts + tag: append_headers_hops_authentication_results_server_root_domain_into_related_hosts + value: '{{{_ingest._value.authentication_results.server.root_domain}}}' + allow_duplicates: false + - foreach: + field: json.headers.hops + if: ctx.json?.headers?.hops instanceof List + processor: + convert: + field: _ingest._value.index + tag: convert_headers_hops_index_to_long + type: long + ignore_missing: true + on_failure: + - remove: + field: _ingest._value.index + ignore_missing: true + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - foreach: + field: json.headers.hops + if: ctx.json?.headers?.hops instanceof List + processor: + date: + field: _ingest._value.received.time + tag: date_headers_hops_received_time + target_field: _ingest._value.received.time + formats: + - ISO8601 + on_failure: + - remove: + field: _ingest._value.received.time + ignore_missing: true + - foreach: + field: json.headers.hops + if: ctx.json?.headers?.hops instanceof List + processor: + convert: + field: _ingest._value.received_spf.client_ip.ip + tag: convert_headers_hops_received_spf_client_ip_ip_to_ip + type: ip + ignore_missing: true + on_failure: + - remove: + field: _ingest._value.received_spf.client_ip.ip + ignore_missing: true + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - foreach: + field: json.headers.hops + if: ctx.json?.headers?.hops instanceof List + processor: + append: + field: related.ip + tag: append_headers_hops_received_spf_client_ip_ip_into_related_ip + value: '{{{_ingest._value.received_spf.client_ip.ip}}}' + allow_duplicates: false + - foreach: + field: json.headers.hops + if: ctx.json?.headers?.hops instanceof List + processor: + append: + field: related.hosts + tag: append_headers_hops_received_spf_helo_domain_into_related_hosts + value: '{{{_ingest._value.received_spf.helo.domain}}}' + allow_duplicates: false + - foreach: + field: json.headers.hops + if: ctx.json?.headers?.hops instanceof List + processor: + append: + field: related.hosts + tag: append_headers_hops_received_spf_helo_root_domain_into_related_hosts + value: '{{{_ingest._value.received_spf.helo.root_domain}}}' + allow_duplicates: false + - foreach: + field: json.headers.hops + if: ctx.json?.headers?.hops instanceof List + processor: + convert: + field: _ingest._value.received_spf.helo.valid + tag: convert_headers_hops_received_spf_helo_valid_to_boolean + type: boolean + ignore_missing: true + on_failure: + - remove: + field: _ingest._value.received_spf.helo.valid + ignore_missing: true + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - foreach: + field: json.headers.hops + if: ctx.json?.headers?.hops instanceof List + processor: + append: + field: related.hosts + tag: append_headers_hops_received_spf_server_domain_into_related_hosts + value: '{{{_ingest._value.received_spf.server.domain}}}' + allow_duplicates: false + - foreach: + field: json.headers.hops + if: ctx.json?.headers?.hops instanceof List + processor: + append: + field: related.hosts + tag: append_headers_hops_received_spf_server_root_domain_into_related_hosts + value: '{{{_ingest._value.received_spf.server.root_domain}}}' + allow_duplicates: false + - foreach: + field: json.headers.hops + if: ctx.json?.headers?.hops instanceof List + processor: + convert: + field: _ingest._value.received_spf.server.valid + tag: convert_headers_hops_received_spf_server_valid_to_boolean + type: boolean + ignore_missing: true + on_failure: + - remove: + field: _ingest._value.received_spf.server.valid + ignore_missing: true + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - foreach: + field: json.headers.hops + if: ctx.json?.headers?.hops instanceof List + processor: + convert: + field: _ingest._value.authentication_results.spf_details.client_ip.ip + tag: convert_headers_hops_authentication_results_spf_details_client_ip_ip_to_ip + type: ip + ignore_missing: true + on_failure: + - remove: + field: _ingest._value.authentication_results.spf_details.client_ip.ip + ignore_missing: true + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - foreach: + field: json.headers.hops + if: ctx.json?.headers?.hops instanceof List + processor: + append: + field: related.ip + tag: append_headers_hops_authentication_results_spf_details_client_ip_ip_into_related_ip + value: '{{{_ingest._value.authentication_results.spf_details.client_ip.ip}}}' + allow_duplicates: false + - foreach: + field: json.headers.hops + if: ctx.json?.headers?.hops instanceof List + processor: + append: + field: related.hosts + tag: append_headers_hops_authentication_results_spf_details_helo_domain_into_related_hosts + value: '{{{_ingest._value.authentication_results.spf_details.helo.domain}}}' + allow_duplicates: false + - foreach: + field: json.headers.hops + if: ctx.json?.headers?.hops instanceof List + processor: + append: + field: related.hosts + tag: append_headers_hops_authentication_results_spf_details_helo_root_domain_into_related_hosts + value: '{{{_ingest._value.authentication_results.spf_details.helo.root_domain}}}' + allow_duplicates: false + - foreach: + field: json.headers.hops + if: ctx.json?.headers?.hops instanceof List + processor: + convert: + field: _ingest._value.authentication_results.spf_details.helo.valid + tag: convert_headers_hops_authentication_results_spf_details_helo_valid_to_boolean + type: boolean + ignore_missing: true + on_failure: + - remove: + field: _ingest._value.authentication_results.spf_details.helo.valid + ignore_missing: true + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - foreach: + field: json.headers.hops + if: ctx.json?.headers?.hops instanceof List + processor: + append: + field: related.hosts + tag: append_headers_hops_authentication_results_spf_details_server_domain_into_related_hosts + value: '{{{_ingest._value.authentication_results.spf_details.server.domain}}}' + allow_duplicates: false + - foreach: + field: json.headers.hops + if: ctx.json?.headers?.hops instanceof List + processor: + append: + field: related.hosts + tag: append_headers_hops_authentication_results_spf_details_server_root_domain_into_related_hosts + value: '{{{_ingest._value.authentication_results.spf_details.server.root_domain}}}' + allow_duplicates: false + - foreach: + field: json.headers.hops + if: ctx.json?.headers?.hops instanceof List + processor: + convert: + field: _ingest._value.authentication_results.spf_details.server.valid + tag: convert_headers_hops_authentication_results_spf_details_server_valid_to_boolean + type: boolean + ignore_missing: true + on_failure: + - remove: + field: _ingest._value.authentication_results.spf_details.server.valid + ignore_missing: true + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - rename: + field: json.headers.hops + tag: rename_headers_hops + target_field: sublime_security.email_message.headers.hops + ignore_missing: true + - rename: + field: json.headers.in_reply_to + tag: rename_headers_in_reply_to + target_field: sublime_security.email_message.headers.in_reply_to + ignore_missing: true + - foreach: + field: json.headers.ips + if: ctx.json?.headers?.ips instanceof List + processor: + append: + field: related.ip + tag: append_headers_ips_ip_into_related_ip + value: '{{{_ingest._value.ip}}}' + allow_duplicates: false + - rename: + field: json.headers.ips + tag: rename_headers_ips + target_field: sublime_security.email_message.headers.ips + ignore_missing: true + - rename: + field: json.headers.mailer + tag: rename_headers_mailer + target_field: sublime_security.email_message.headers.mailer + ignore_missing: true + - set: + field: email.x_mailer + tag: set_email_x_mailer_from_email_message_headers_mailer + copy_from: sublime_security.email_message.headers.mailer + ignore_empty_value: true + - user_agent: + field: email.x_mailer + ignore_missing: true + - rename: + field: json.headers.message_id + tag: rename_headers_message_id + target_field: sublime_security.email_message.headers.message_id + ignore_missing: true + - set: + field: email.message_id + tag: set_email_message_id_from_email_message_headers_message_id + copy_from: sublime_security.email_message.headers.message_id + ignore_empty_value: true + - rename: + field: json.headers.references + tag: rename_headers_references + target_field: sublime_security.email_message.headers.references + ignore_missing: true + - foreach: + field: json.headers.reply_to + if: ctx.json?.headers?.reply_to instanceof List + processor: + append: + field: related.hosts + tag: append_headers_reply_to_email_domain_domain_into_related_hosts + value: '{{{_ingest._value.email.domain.domain}}}' + allow_duplicates: false + - foreach: + field: json.headers.reply_to + if: ctx.json?.headers?.reply_to instanceof List + processor: + append: + field: related.hosts + tag: append_headers_reply_to_email_domain_root_domain_into_related_hosts + value: '{{{_ingest._value.email.domain.root_domain}}}' + allow_duplicates: false + - foreach: + field: json.headers.reply_to + if: ctx.json?.headers?.reply_to instanceof List + processor: + convert: + field: _ingest._value.email.domain.valid + tag: convert_headers_reply_to_email_domain_valid_to_boolean + type: boolean + ignore_missing: true + on_failure: + - remove: + field: _ingest._value.email.domain.valid + ignore_missing: true + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - foreach: + field: json.headers.reply_to + if: ctx.json?.headers?.reply_to instanceof List + processor: + append: + field: email.reply_to.address + tag: append_headers_reply_to_email_email_into_email_reply_to_address + value: '{{{_ingest._value.email.email}}}' + allow_duplicates: false + - foreach: + field: json.headers.reply_to + if: ctx.json?.headers?.reply_to instanceof List + processor: + append: + field: related.user + tag: append_headers_reply_to_email_email_into_related_user + value: '{{{_ingest._value.email.email}}}' + allow_duplicates: false + - foreach: + field: json.headers.reply_to + if: ctx.json?.headers?.reply_to instanceof List + processor: + rename: + field: _ingest._value.email.email + tag: rename_headers_reply_to_email_email_to_value + target_field: _ingest._value.email.value + ignore_missing: true + - rename: + field: json.headers.reply_to + tag: rename_headers_reply_to + target_field: sublime_security.email_message.headers.reply_to + ignore_missing: true + - rename: + field: json.headers.return_path.domain.domain + tag: rename_headers_return_path_domain_domain + target_field: sublime_security.email_message.headers.return_path.domain.domain + ignore_missing: true + - append: + field: related.hosts + tag: append_headers_return_path_domain_domain_into_related_hosts + value: '{{{sublime_security.email_message.headers.return_path.domain.domain}}}' + allow_duplicates: false + - rename: + field: json.headers.return_path.domain.punycode + tag: rename_headers_return_path_domain_punycode + target_field: sublime_security.email_message.headers.return_path.domain.punycode + ignore_missing: true + - rename: + field: json.headers.return_path.domain.root_domain + tag: rename_headers_return_path_domain_root_domain + target_field: sublime_security.email_message.headers.return_path.domain.root_domain + ignore_missing: true + - append: + field: related.hosts + tag: append_headers_return_path_domain_root_domain_into_related_hosts + value: '{{{sublime_security.email_message.headers.return_path.domain.root_domain}}}' + allow_duplicates: false + - rename: + field: json.headers.return_path.domain.sld + tag: rename_headers_return_path_domain_sld + target_field: sublime_security.email_message.headers.return_path.domain.sld + ignore_missing: true + - rename: + field: json.headers.return_path.domain.subdomain + tag: rename_headers_return_path_domain_subdomain + target_field: sublime_security.email_message.headers.return_path.domain.subdomain + ignore_missing: true + - rename: + field: json.headers.return_path.domain.tld + tag: rename_headers_return_path_domain_tld + target_field: sublime_security.email_message.headers.return_path.domain.tld + ignore_missing: true + - convert: + field: json.headers.return_path.domain.valid + tag: convert_headers_return_path_domain_valid_to_boolean + target_field: sublime_security.email_message.headers.return_path.domain.valid + type: boolean + ignore_missing: true + on_failure: + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - rename: + field: json.headers.return_path.email + tag: rename_headers_return_path_email + target_field: sublime_security.email_message.headers.return_path.email + ignore_missing: true + - rename: + field: json.headers.return_path.local_part + tag: rename_headers_return_path_local_part + target_field: sublime_security.email_message.headers.return_path.local_part + ignore_missing: true + - rename: + field: json.headers.x_authenticated_domain.domain + tag: rename_headers_x_authenticated_domain_domain + target_field: sublime_security.email_message.headers.x_authenticated_domain.domain + ignore_missing: true + - append: + field: related.hosts + tag: append_headers_x_authenticated_domain_domain_into_related_hosts + value: '{{{sublime_security.email_message.headers.x_authenticated_domain.domain}}}' + allow_duplicates: false + - rename: + field: json.headers.x_authenticated_domain.punycode + tag: rename_headers_x_authenticated_domain_punycode + target_field: sublime_security.email_message.headers.x_authenticated_domain.punycode + ignore_missing: true + - rename: + field: json.headers.x_authenticated_domain.root_domain + tag: rename_headers_x_authenticated_domain_root_domain + target_field: sublime_security.email_message.headers.x_authenticated_domain.root_domain + ignore_missing: true + - append: + field: related.hosts + tag: append_headers_x_authenticated_domain_root_domain_into_related_hosts + value: '{{{sublime_security.email_message.headers.x_authenticated_domain.root_domain}}}' + allow_duplicates: false + - rename: + field: json.headers.x_authenticated_domain.sld + tag: rename_headers_x_authenticated_domain_sld + target_field: sublime_security.email_message.headers.x_authenticated_domain.sld + ignore_missing: true + - rename: + field: json.headers.x_authenticated_domain.subdomain + tag: rename_headers_x_authenticated_domain_subdomain + target_field: sublime_security.email_message.headers.x_authenticated_domain.subdomain + ignore_missing: true + - rename: + field: json.headers.x_authenticated_domain.tld + tag: rename_headers_x_authenticated_domain_tld + target_field: sublime_security.email_message.headers.x_authenticated_domain.tld + ignore_missing: true + - convert: + field: json.headers.x_authenticated_domain.valid + tag: convert_headers_x_authenticated_domain_valid_to_boolean + target_field: sublime_security.email_message.headers.x_authenticated_domain.valid + type: boolean + ignore_missing: true + on_failure: + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - rename: + field: json.headers.x_authenticated_sender.domain.domain + tag: rename_headers_x_authenticated_sender_domain_domain + target_field: sublime_security.email_message.headers.x_authenticated_sender.domain.domain + ignore_missing: true + - append: + field: related.hosts + tag: append_headers_x_authenticated_sender_domain_domain_into_related_hosts + value: '{{{sublime_security.email_message.headers.x_authenticated_sender.domain.domain}}}' + allow_duplicates: false + - rename: + field: json.headers.x_authenticated_sender.domain.punycode + tag: rename_headers_x_authenticated_sender_domain_punycode + target_field: sublime_security.email_message.headers.x_authenticated_sender.domain.punycode + ignore_missing: true + - rename: + field: json.headers.x_authenticated_sender.domain.root_domain + tag: rename_headers_x_authenticated_sender_domain_root_domain + target_field: sublime_security.email_message.headers.x_authenticated_sender.domain.root_domain + ignore_missing: true + - append: + field: related.hosts + tag: append_headers_x_authenticated_sender_domain_root_domain_into_related_hosts + value: '{{{sublime_security.email_message.headers.x_authenticated_sender.domain.root_domain}}}' + allow_duplicates: false + - rename: + field: json.headers.x_authenticated_sender.domain.sld + tag: rename_headers_x_authenticated_sender_domain_sld + target_field: sublime_security.email_message.headers.x_authenticated_sender.domain.sld + ignore_missing: true + - rename: + field: json.headers.x_authenticated_sender.domain.subdomain + tag: rename_headers_x_authenticated_sender_domain_subdomain + target_field: sublime_security.email_message.headers.x_authenticated_sender.domain.subdomain + ignore_missing: true + - rename: + field: json.headers.x_authenticated_sender.domain.tld + tag: rename_headers_x_authenticated_sender_domain_tld + target_field: sublime_security.email_message.headers.x_authenticated_sender.domain.tld + ignore_missing: true + - convert: + field: json.headers.x_authenticated_sender.domain.valid + tag: convert_headers_x_authenticated_sender_domain_valid_to_boolean + target_field: sublime_security.email_message.headers.x_authenticated_sender.domain.valid + type: boolean + ignore_missing: true + on_failure: + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - rename: + field: json.headers.x_authenticated_sender.email + tag: rename_headers_x_authenticated_sender_email + target_field: sublime_security.email_message.headers.x_authenticated_sender.email + ignore_missing: true + - rename: + field: json.headers.x_authenticated_sender.local_part + tag: rename_headers_x_authenticated_sender_local_part + target_field: sublime_security.email_message.headers.x_authenticated_sender.local_part + ignore_missing: true + - convert: + field: json.headers.x_client_ip.ip + tag: convert_headers_x_client_ip_ip_to_ip + target_field: sublime_security.email_message.headers.x_client_ip.ip + type: ip + ignore_missing: true + if: ctx.json?.headers?.x_client_ip?.ip != null && ctx.json.headers.x_client_ip.ip != '' + on_failure: + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - set: + field: source.ip + tag: set_source_ip_from_headers_x_client_ip_ip + copy_from: sublime_security.email_message.headers.x_client_ip.ip + ignore_empty_value: true + - append: + field: related.ip + tag: append_headers_x_client_ip_ip_into_related_ip + value: '{{{sublime_security.email_message.headers.x_client_ip.ip}}}' + allow_duplicates: false + - convert: + field: json.headers.x_originating_ip.ip + tag: convert_headers_x_originating_ip_ip_to_ip + target_field: sublime_security.email_message.headers.x_originating_ip.ip + type: ip + ignore_missing: true + if: ctx.json?.headers?.x_originating_ip?.ip != null && ctx.json.headers.x_originating_ip.ip != '' + on_failure: + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - append: + field: related.ip + tag: append_headers_x_originating_ip_ip_into_related_ip + value: '{{{sublime_security.email_message.headers.x_originating_ip.ip}}}' + allow_duplicates: false + - rename: + field: json.headers.x_secure_server_account + tag: rename_headers_x_secure_server_account + target_field: sublime_security.email_message.headers.x_secure_server_account + ignore_missing: true + - rename: + field: json.headers.x_sender.domain.domain + tag: rename_headers_x_sender_domain_domain + target_field: sublime_security.email_message.headers.x_sender.domain.domain + ignore_missing: true + - append: + field: related.hosts + tag: append_headers_x_sender_domain_domain_into_related_hosts + value: '{{{sublime_security.email_message.headers.x_sender.domain.domain}}}' + allow_duplicates: false + - rename: + field: json.headers.x_sender.domain.punycode + tag: rename_headers_x_sender_domain_punycode + target_field: sublime_security.email_message.headers.x_sender.domain.punycode + ignore_missing: true + - rename: + field: json.headers.x_sender.domain.root_domain + tag: rename_headers_x_sender_domain_root_domain + target_field: sublime_security.email_message.headers.x_sender.domain.root_domain + ignore_missing: true + - append: + field: related.hosts + tag: append_headers_x_sender_domain_root_domain_into_related_hosts + value: '{{{sublime_security.email_message.headers.x_sender.domain.root_domain}}}' + allow_duplicates: false + - rename: + field: json.headers.x_sender.domain.sld + tag: rename_headers_x_sender_domain_sld + target_field: sublime_security.email_message.headers.x_sender.domain.sld + ignore_missing: true + - rename: + field: json.headers.x_sender.domain.subdomain + tag: rename_headers_x_sender_domain_subdomain + target_field: sublime_security.email_message.headers.x_sender.domain.subdomain + ignore_missing: true + - rename: + field: json.headers.x_sender.domain.tld + tag: rename_headers_x_sender_domain_tld + target_field: sublime_security.email_message.headers.x_sender.domain.tld + ignore_missing: true + - convert: + field: json.headers.x_sender.domain.valid + tag: convert_headers_x_sender_domain_valid_to_boolean + target_field: sublime_security.email_message.headers.x_sender.domain.valid + type: boolean + ignore_missing: true + on_failure: + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - rename: + field: json.headers.x_sender.email + tag: rename_headers_x_sender_email + target_field: sublime_security.email_message.headers.x_sender.email + ignore_missing: true + - rename: + field: json.headers.x_sender.local_part + tag: rename_headers_x_sender_local_part + target_field: sublime_security.email_message.headers.x_sender.local_part + ignore_missing: true + - rename: + field: json.mailbox.display_name + tag: rename_mailbox_display_name + target_field: sublime_security.email_message.mailbox.display_name + ignore_missing: true + - rename: + field: json.mailbox.email.domain.domain + tag: rename_mailbox_email_domain_domain + target_field: sublime_security.email_message.mailbox.email.domain.domain + ignore_missing: true + - append: + field: related.hosts + tag: append_mailbox_email_domain_domain_into_related_hosts + value: '{{{sublime_security.email_message.mailbox.email.domain.domain}}}' + allow_duplicates: false + - rename: + field: json.mailbox.email.domain.punycode + tag: rename_mailbox_email_domain_punycode + target_field: sublime_security.email_message.mailbox.email.domain.punycode + ignore_missing: true + - rename: + field: json.mailbox.email.domain.root_domain + tag: rename_mailbox_email_domain_root_domain + target_field: sublime_security.email_message.mailbox.email.domain.root_domain + ignore_missing: true + - append: + field: related.hosts + tag: append_mailbox_email_domain_root_domain_into_related_hosts + value: '{{{sublime_security.email_message.mailbox.email.domain.root_domain}}}' + allow_duplicates: false + - rename: + field: json.mailbox.email.domain.sld + tag: rename_mailbox_email_domain_sld + target_field: sublime_security.email_message.mailbox.email.domain.sld + ignore_missing: true + - rename: + field: json.mailbox.email.domain.subdomain + tag: rename_mailbox_email_domain_subdomain + target_field: sublime_security.email_message.mailbox.email.domain.subdomain + ignore_missing: true + - rename: + field: json.mailbox.email.domain.tld + tag: rename_mailbox_email_domain_tld + target_field: sublime_security.email_message.mailbox.email.domain.tld + ignore_missing: true + - convert: + field: json.mailbox.email.domain.valid + tag: convert_mailbox_email_domain_valid_to_boolean + target_field: sublime_security.email_message.mailbox.email.domain.valid + type: boolean + ignore_missing: true + on_failure: + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - rename: + field: json.mailbox.email.email + tag: rename_mailbox_email_email + target_field: sublime_security.email_message.mailbox.email.value + ignore_missing: true + - append: + field: related.user + tag: append_mailbox_email_value_into_related_user + value: '{{{sublime_security.email_message.mailbox.email.value}}}' + allow_duplicates: false + - rename: + field: json.mailbox.email.local_part + tag: rename_mailbox_email_local_part + target_field: sublime_security.email_message.mailbox.email.local_part + ignore_missing: true + - rename: + field: json._meta.canonical_id + tag: rename_meta_canonical_id + target_field: sublime_security.email_message.meta.canonical_id + ignore_missing: true + - date: + field: json._meta.created_at + tag: date_meta_created_at + target_field: sublime_security.email_message.meta.created_at + formats: + - ISO8601 + if: ctx.json?._meta?.created_at != null && ctx.json._meta.created_at != '' + on_failure: + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - set: + field: '@timestamp' + tag: set_@timestamp_from_email_message_meta_created_at + copy_from: sublime_security.email_message.meta.created_at + ignore_empty_value: true + - date: + field: json._meta.effective_at + tag: date_meta_effective_at + target_field: sublime_security.email_message.meta.effective_at + formats: + - ISO8601 + if: ctx.json?._meta?.effective_at != null && ctx.json._meta.effective_at != '' + on_failure: + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - rename: + field: json._meta.id + tag: rename_meta_id + target_field: sublime_security.email_message.meta.id + ignore_missing: true + - set: + field: event.id + tag: set_event_id_from_meta_id + copy_from: sublime_security.email_message.meta.id + ignore_empty_value: true + - foreach: + field: json.recipients.bcc + if: ctx.json?.recipients?.bcc instanceof List + processor: + append: + field: related.hosts + tag: append_recipients_bcc_email_domain_domain_into_related_hosts + value: '{{{_ingest._value.email.domain.domain}}}' + allow_duplicates: false + - foreach: + field: json.recipients.bcc + if: ctx.json?.recipients?.bcc instanceof List + processor: + append: + field: related.hosts + tag: append_recipients_bcc_email_domain_root_domain_into_related_hosts + value: '{{{_ingest._value.email.domain.root_domain}}}' + allow_duplicates: false + - foreach: + field: json.recipients.bcc + if: ctx.json?.recipients?.bcc instanceof List + processor: + convert: + field: _ingest._value.email.domain.valid + tag: convert_recipients_bcc_email_domain_valid_to_boolean + type: boolean + ignore_missing: true + on_failure: + - remove: + field: _ingest._value.email.domain.valid + ignore_missing: true + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - foreach: + field: json.recipients.bcc + if: ctx.json?.recipients?.bcc instanceof List + processor: + append: + field: email.bcc.address + tag: append_recipients_bcc_email_email_into_email_bcc_address + value: '{{{_ingest._value.email.email}}}' + allow_duplicates: false + - foreach: + field: json.recipients.bcc + if: ctx.json?.recipients?.bcc instanceof List + processor: + append: + field: related.user + tag: append_recipients_bcc_email_email_into_related_user + value: '{{{_ingest._value.email.email}}}' + allow_duplicates: false + - foreach: + field: json.recipients.bcc + if: ctx.json?.recipients?.bcc instanceof List + processor: + rename: + field: _ingest._value.email.email + tag: rename_recipients_bcc_email_email_to_value + target_field: _ingest._value.email.value + ignore_missing: true + - rename: + field: json.recipients.bcc + tag: rename_recipients_bcc + target_field: sublime_security.email_message.recipients.bcc + ignore_missing: true + - foreach: + field: json.recipients.cc + if: ctx.json?.recipients?.cc instanceof List + processor: + append: + field: related.hosts + tag: append_recipients_cc_email_domain_domain_into_related_hosts + value: '{{{_ingest._value.email.domain.domain}}}' + allow_duplicates: false + - foreach: + field: json.recipients.cc + if: ctx.json?.recipients?.cc instanceof List + processor: + append: + field: related.hosts + tag: append_recipients_cc_email_domain_root_domain_into_related_hosts + value: '{{{_ingest._value.email.domain.root_domain}}}' + allow_duplicates: false + - foreach: + field: json.recipients.cc + if: ctx.json?.recipients?.cc instanceof List + processor: + convert: + field: _ingest._value.email.domain.valid + tag: convert_recipients_cc_email_domain_valid_to_boolean + type: boolean + ignore_missing: true + on_failure: + - remove: + field: _ingest._value.email.domain.valid + ignore_missing: true + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - foreach: + field: json.recipients.cc + if: ctx.json?.recipients?.cc instanceof List + processor: + append: + field: email.cc.address + tag: append_recipients_cc_email_email_into_email_cc_address + value: '{{{_ingest._value.email.email}}}' + allow_duplicates: false + - foreach: + field: json.recipients.cc + if: ctx.json?.recipients?.cc instanceof List + processor: + append: + field: related.user + tag: append_recipients_cc_email_email_into_related_user + value: '{{{_ingest._value.email.email}}}' + allow_duplicates: false + - foreach: + field: json.recipients.cc + if: ctx.json?.recipients?.cc instanceof List + processor: + rename: + field: _ingest._value.email.email + tag: rename_recipients_cc_email_email_to_value + target_field: _ingest._value.email.value + ignore_missing: true + - rename: + field: json.recipients.cc + tag: rename_recipients_cc + target_field: sublime_security.email_message.recipients.cc + ignore_missing: true + - foreach: + field: json.recipients.to + if: ctx.json?.recipients?.to instanceof List + processor: + append: + field: related.hosts + tag: append_recipients_to_email_domain_domain_into_related_hosts + value: '{{{_ingest._value.email.domain.domain}}}' + allow_duplicates: false + - foreach: + field: json.recipients.to + if: ctx.json?.recipients?.to instanceof List + processor: + append: + field: related.hosts + tag: append_recipients_to_email_domain_root_domain_into_related_hosts + value: '{{{_ingest._value.email.domain.root_domain}}}' + allow_duplicates: false + - foreach: + field: json.recipients.to + if: ctx.json?.recipients?.to instanceof List + processor: + convert: + field: _ingest._value.email.domain.valid + tag: convert_recipients_to_email_domain_valid_to_boolean + type: boolean + ignore_missing: true + on_failure: + - remove: + field: _ingest._value.email.domain.valid + ignore_missing: true + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - foreach: + field: json.recipients.to + if: ctx.json?.recipients?.to instanceof List + processor: + append: + field: email.to.address + tag: append_recipients_to_email_email_into_email_to_address + value: '{{{_ingest._value.email.email}}}' + allow_duplicates: false + - foreach: + field: json.recipients.to + if: ctx.json?.recipients?.to instanceof List + processor: + append: + field: related.user + tag: append_recipients_to_email_email_into_related_user + value: '{{{_ingest._value.email.email}}}' + allow_duplicates: false + - foreach: + field: json.recipients.to + if: ctx.json?.recipients?.to instanceof List + processor: + rename: + field: _ingest._value.email.email + tag: rename_recipients_to_email_email_to_value + target_field: _ingest._value.email.value + ignore_missing: true + - rename: + field: json.recipients.to + tag: rename_recipients_to + target_field: sublime_security.email_message.recipients.to + ignore_missing: true + - rename: + field: json.sender.display_name + tag: rename_sender_display_name + target_field: sublime_security.email_message.sender.display_name + ignore_missing: true + - rename: + field: json.sender.email.domain.domain + tag: rename_sender_email_domain_domain + target_field: sublime_security.email_message.sender.email.domain.domain + ignore_missing: true + - append: + field: related.hosts + tag: append_sender_email_domain_domain_into_related_hosts + value: '{{{sublime_security.email_message.sender.email.domain.domain}}}' + allow_duplicates: false + - set: + field: source.domain + tag: set_source_domain_from_sender_email_domain_domain + copy_from: sublime_security.email_message.sender.email.domain.domain + ignore_empty_value: true + - rename: + field: json.sender.email.domain.punycode + tag: rename_sender_email_domain_punycode + target_field: sublime_security.email_message.sender.email.domain.punycode + ignore_missing: true + - rename: + field: json.sender.email.domain.root_domain + tag: rename_sender_email_domain_root_domain + target_field: sublime_security.email_message.sender.email.domain.root_domain + ignore_missing: true + - append: + field: related.hosts + tag: append_sender_email_domain_root_domain_into_related_hosts + value: '{{{sublime_security.email_message.sender.email.domain.root_domain}}}' + allow_duplicates: false + - rename: + field: json.sender.email.domain.sld + tag: rename_sender_email_domain_sld + target_field: sublime_security.email_message.sender.email.domain.sld + ignore_missing: true + - rename: + field: json.sender.email.domain.subdomain + tag: rename_sender_email_domain_subdomain + target_field: sublime_security.email_message.sender.email.domain.subdomain + ignore_missing: true + - set: + field: source.subdomain + tag: set_source_subdomain_from_sender_email_domain_subdomain + copy_from: sublime_security.email_message.sender.email.domain.subdomain + ignore_empty_value: true + - rename: + field: json.sender.email.domain.tld + tag: rename_sender_email_domain_tld + target_field: sublime_security.email_message.sender.email.domain.tld + ignore_missing: true + - set: + field: source.top_level_domain + tag: set_source_top_level_domain_from_sender_email_domain_tld + copy_from: sublime_security.email_message.sender.email.domain.tld + ignore_empty_value: true + - convert: + field: json.sender.email.domain.valid + tag: convert_sender_email_domain_valid_to_boolean + target_field: sublime_security.email_message.sender.email.domain.valid + type: boolean + ignore_missing: true + on_failure: + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - rename: + field: json.sender.email.email + tag: rename_sender_email_email + target_field: sublime_security.email_message.sender.email.value + ignore_missing: true + - append: + field: email.from.address + tag: append_sublime_security_email_message_sender_email_value_into_email_from_address + value: '{{{sublime_security.email_message.sender.email.value}}}' + allow_duplicates: false + if: ctx.sublime_security?.email_message?.sender?.email?.value != null + - append: + field: related.user + tag: append_sender_email_value_into_related_user + value: '{{{sublime_security.email_message.sender.email.value}}}' + allow_duplicates: false + - rename: + field: json.sender.email.local_part + tag: rename_sender_email_local_part + target_field: sublime_security.email_message.sender.email.local_part + ignore_missing: true + - rename: + field: json.subject.subject + tag: rename_subject_subject + target_field: sublime_security.email_message.subject.subject + ignore_missing: true + - set: + field: email.subject + tag: set_email_subject_from_email_message_subject_subject + copy_from: sublime_security.email_message.subject.subject + ignore_empty_value: true + - convert: + field: json.type.inbound + tag: convert_type_inbound_to_boolean + target_field: sublime_security.email_message.type.inbound + type: boolean + ignore_missing: true + on_failure: + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - set: + field: email.direction + tag: set_email_direction_inbound_from_type_inbound + value: inbound + if: ctx.sublime_security?.email_message?.type?.inbound == true + ignore_empty_value: true + - convert: + field: json.type.internal + tag: convert_type_internal_to_boolean + target_field: sublime_security.email_message.type.internal + type: boolean + ignore_missing: true + on_failure: + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - set: + field: email.direction + tag: set_email_direction_internal_from_type_internal + value: internal + if: ctx.sublime_security?.email_message?.type?.internal == true + ignore_empty_value: true + - convert: + field: json.type.outbound + tag: convert_type_outbound_to_boolean + target_field: sublime_security.email_message.type.outbound + type: boolean + ignore_missing: true + on_failure: + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - set: + field: email.direction + tag: set_email_direction_outbound_from_type_outbound + value: outbound + if: ctx.sublime_security?.email_message?.type?.outbound == true + ignore_empty_value: true + - foreach: + field: sublime_security.email_message.attachments + if: ctx.sublime_security?.email_message?.attachments instanceof List + processor: + remove: + field: + - _ingest._value.content.type + - _ingest._value.file.extension + - _ingest._value.file.name + - _ingest._value.md5 + - _ingest._value.sha1 + - _ingest._value.sha256 + - _ingest._value.size + tag: remove_custom_duplicate_fields_from_sublime_security_email_message_attachments + ignore_missing: true + if: ctx.tags == null || !(ctx.tags.contains('preserve_duplicate_custom_fields')) + - foreach: + field: sublime_security.email_message.body.links + if: ctx.sublime_security?.email_message?.body?.links instanceof List + processor: + remove: + field: + - _ingest._value.href_url.domain.domain + - _ingest._value.href_url.domain.subdomain + - _ingest._value.href_url.domain.tld + - _ingest._value.href_url.fragment + - _ingest._value.href_url.password + - _ingest._value.href_url.path + - _ingest._value.href_url.port + - _ingest._value.href_url.query_params + - _ingest._value.href_url.scheme + - _ingest._value.href_url.url + - _ingest._value.href_url.username + tag: remove_custom_duplicate_fields_from_sublime_security_email_message_body_links + ignore_missing: true + if: ctx.tags == null || !(ctx.tags.contains('preserve_duplicate_custom_fields')) + - foreach: + field: sublime_security.email_message.headers.reply_to + if: ctx.sublime_security?.email_message?.headers?.reply_to instanceof List + processor: + remove: + field: _ingest._value.email.value + tag: remove_custom_duplicate_fields_from_sublime_security_email_message_headers_reply_to + ignore_missing: true + if: ctx.tags == null || !(ctx.tags.contains('preserve_duplicate_custom_fields')) + - foreach: + field: sublime_security.email_message.recipients.bcc + if: ctx.sublime_security?.email_message?.recipients?.bcc instanceof List + processor: + remove: + field: _ingest._value.email.value + tag: remove_custom_duplicate_fields_from_sublime_security_email_message_recipients_bcc + ignore_missing: true + if: ctx.tags == null || !(ctx.tags.contains('preserve_duplicate_custom_fields')) + - foreach: + field: sublime_security.email_message.recipients.cc + if: ctx.sublime_security?.email_message?.recipients?.cc instanceof List + processor: + remove: + field: _ingest._value.email.value + tag: remove_custom_duplicate_fields_from_sublime_security_email_message_recipients_cc + ignore_missing: true + if: ctx.tags == null || !(ctx.tags.contains('preserve_duplicate_custom_fields')) + - foreach: + field: sublime_security.email_message.recipients.to + if: ctx.sublime_security?.email_message?.recipients?.to instanceof List + processor: + remove: + field: _ingest._value.email.value + tag: remove_custom_duplicate_fields_from_sublime_security_email_message_recipients_to + ignore_missing: true + if: ctx.tags == null || !(ctx.tags.contains('preserve_duplicate_custom_fields')) + - remove: + field: + - sublime_security.email_message.external.created_at + - sublime_security.email_message.headers.delivered_to.domain.domain + - sublime_security.email_message.headers.delivered_to.domain.subdomain + - sublime_security.email_message.headers.delivered_to.domain.tld + - sublime_security.email_message.headers.mailer + - sublime_security.email_message.headers.message_id + - sublime_security.email_message.headers.x_client_ip.ip + - sublime_security.email_message.meta.created_at + - sublime_security.email_message.meta.id + - sublime_security.email_message.sender.email.domain.domain + - sublime_security.email_message.sender.email.domain.subdomain + - sublime_security.email_message.sender.email.domain.tld + - sublime_security.email_message.sender.email.value + - sublime_security.email_message.subject.subject + tag: remove_custom_duplicate_fields + ignore_missing: true + if: ctx.tags == null || !(ctx.tags.contains('preserve_duplicate_custom_fields')) + - remove: + field: json + tag: remove_json + ignore_missing: true + - script: + lang: painless + description: Drops null/empty values recursively. + tag: painless_remove_null + source: |- + boolean drop(Object object) { + if (object == null || object == '') { + return true; + } else if (object instanceof Map) { + ((Map) object).values().removeIf(v -> drop(v)); + return (((Map) object).size() == 0); + } else if (object instanceof List) { + ((List) object).removeIf(v -> drop(v)); + return (((List) object).length == 0); + } + return false; + } + drop(ctx); + - set: + field: event.kind + value: pipeline_error + tag: set_pipeline_error_into_event_kind + if: ctx.error?.message != null +on_failure: + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - set: + field: event.kind + tag: set_pipeline_error_to_event_kind + value: pipeline_error diff --git a/packages/sublime_security/data_stream/email_message/fields/base-fields.yml b/packages/sublime_security/data_stream/email_message/fields/base-fields.yml new file mode 100644 index 00000000000..97435bd3ccc --- /dev/null +++ b/packages/sublime_security/data_stream/email_message/fields/base-fields.yml @@ -0,0 +1,20 @@ +- name: data_stream.type + type: constant_keyword + description: Data stream type. +- name: data_stream.dataset + type: constant_keyword + description: Data stream dataset. +- name: data_stream.namespace + type: constant_keyword + description: Data stream namespace. +- name: event.module + type: constant_keyword + description: Event module. + value: sublime_security +- name: event.dataset + type: constant_keyword + description: Event dataset. + value: sublime_security.email_message +- name: '@timestamp' + type: date + description: Event timestamp. diff --git a/packages/sublime_security/data_stream/email_message/fields/beats.yml b/packages/sublime_security/data_stream/email_message/fields/beats.yml new file mode 100644 index 00000000000..fff1b3f1b6b --- /dev/null +++ b/packages/sublime_security/data_stream/email_message/fields/beats.yml @@ -0,0 +1,24 @@ +- name: input.type + type: keyword + description: Type of filebeat input. +- name: log.offset + type: long + description: Log offset. +- name: aws.s3 + type: group + fields: + - name: bucket + type: group + fields: + - name: name + type: keyword + description: The AWS S3 bucket name. + - name: arn + type: keyword + description: The AWS S3 bucket ARN. + - name: object + type: group + fields: + - name: key + type: keyword + description: The AWS S3 Object key. diff --git a/packages/sublime_security/data_stream/email_message/fields/fields.yml b/packages/sublime_security/data_stream/email_message/fields/fields.yml new file mode 100644 index 00000000000..ebd5e6a90f5 --- /dev/null +++ b/packages/sublime_security/data_stream/email_message/fields/fields.yml @@ -0,0 +1,1222 @@ +- name: sublime_security + type: group + fields: + - name: email_message + type: group + fields: + - name: attachments + type: group + fields: + - name: content + type: group + fields: + - name: id + type: keyword + description: Content-ID extracted from the MIME payload. + - name: transfer_encoding + type: keyword + description: Content-Transfer-Encoding extracted from the MIME payload. + - name: type + type: keyword + description: Content-Type extracted from the MIME payload. + - name: file + type: group + fields: + - name: extension + type: keyword + description: File extension from context such as headers. + - name: name + type: keyword + description: File name. + - name: type + type: keyword + description: File type determined by looking at the magic bytes in the file. + - name: md5 + type: keyword + description: MD5 hash of the raw contents. + - name: raw + type: keyword + description: Base64 encoded source of the file. + - name: sha1 + type: keyword + description: SHA1 hash of the raw contents. + - name: sha256 + type: keyword + description: SHA256 hash of the raw contents. + - name: size + type: long + description: Size of the file in bytes. + - name: body + type: group + fields: + - name: current_thread + type: group + fields: + - name: text + type: keyword + description: The text content from the latest reply/forward in a message thread. This typically excludes content from forwarded messages and warning banners. + - name: html + type: group + fields: + - name: charset + type: keyword + description: charset of the text/[subtype]. + - name: content_transfer_encoding + type: keyword + description: Content-Transfer-Encoding of the text/[subtype]. + - name: display_text + type: keyword + description: Visible text of the HTML document, with invisible characters removed and non-ASCII characters converted to ASCII spaces. + - name: inner_text + type: keyword + description: Inner text of the HTML document that doesn't include HTML tags. + - name: raw + type: keyword + description: Decoded raw content of a body text type (text/[subtype] section). + - name: ips + type: group + fields: + - name: ip + type: ip + description: The raw IP. + - name: links + type: group + fields: + - name: display_text + type: keyword + description: The text of a hyperlink, if it's not a URL. + - name: display_url + type: group + fields: + - name: domain + type: group + fields: + - name: domain + type: keyword + description: The fully qualified domain name (FQDN). This may not always be routable, e.g. when an email address contains a domain that is just a TLD with no SLD, e.g. foo@WIN-bar. + - name: punycode + type: keyword + description: Interpreted punycode if the domain starts with xn--. For example, if 'domain' is 'xn--ublimesecurity-4xc.com' then 'punycode' is śublimesecurity.com. + - name: root_domain + type: keyword + description: The root domain, including the TLD. + - name: sld + type: keyword + description: Second-level domain, e.g. 'windows' for the domain 'windows.net'. + - name: subdomain + type: keyword + description: Subdomain, e.g. 'drive' for the domain 'drive.google.com'. + - name: tld + type: keyword + description: The domain's top-level domain. E.g. the TLD of google.com is 'com'. + - name: valid + type: boolean + description: Whether the domain is valid. + - name: fragment + type: keyword + description: 'Fragment identifier; the text following the # in the href_url (also called the anchor tag).' + - name: password + type: keyword + description: The password specified before the domain name. + - name: path + type: keyword + description: Everything after the TLD and before the query parameters. + - name: port + type: long + description: The port used for the href_url. If no explicit port is set, the port will be inferred from the protocol. + - name: query_params + type: keyword + description: The query parameters of the href_url. + - name: rewrite + type: group + fields: + - name: encoders + type: keyword + description: List of detected URL rewrite encoders while unraveling the URL. + - name: original + type: keyword + description: Original URL without any unraveling URL rewrites. + - name: scheme + type: keyword + description: Protocol for the href_url request, e.g. http. + - name: url + type: keyword + description: Full URL. + - name: username + type: keyword + description: The username specified before the domain name of the href_url. + - name: href_url + type: group + fields: + - name: domain + type: group + fields: + - name: domain + type: keyword + description: The fully qualified domain name (FQDN). This may not always be routable, e.g. when an email address contains a domain that is just a TLD with no SLD, e.g. foo@WIN-bar. + - name: punycode + type: keyword + description: Interpreted punycode if the domain starts with xn--. For example, if 'domain' is 'xn--ublimesecurity-4xc.com' then 'punycode' is śublimesecurity.com. + - name: root_domain + type: keyword + description: The root domain, including the TLD. + - name: sld + type: keyword + description: Second-level domain, e.g. 'windows' for the domain 'windows.net'. + - name: subdomain + type: keyword + description: Subdomain, e.g. 'drive' for the domain 'drive.google.com'. + - name: tld + type: keyword + description: The domain's top-level domain. E.g. the TLD of google.com is 'com'. + - name: valid + type: boolean + description: Whether the domain is valid. + - name: fragment + type: keyword + description: 'Fragment identifier; the text following the # in the href_url (also called the anchor tag).' + - name: password + type: keyword + description: The password specified before the domain name. + - name: path + type: keyword + description: Everything after the TLD and before the query parameters. + - name: port + type: long + description: The port used for the href_url. If no explicit port is set, the port will be inferred from the protocol. + - name: query_params + type: keyword + description: The query parameters of the href_url. + - name: rewrite + type: group + fields: + - name: encoders + type: keyword + description: List of detected URL rewrite encoders while unraveling the URL. + - name: original + type: keyword + description: Original URL without any unraveling URL rewrites. + - name: scheme + type: keyword + description: Protocol for the href_url request, e.g. http. + - name: url + type: keyword + description: Full URL. + - name: username + type: keyword + description: The username specified before the domain name of the href_url. + - name: mismatched + type: boolean + description: Whether the display URL and href URL root domains are mismatched (i.e. .href_url.domain.root_domain != .display_url.domain.root_domain, where both are not null and valid domains). + - name: plain + type: group + fields: + - name: charset + type: keyword + description: charset of the text/[subtype]. + - name: content_transfer_encoding + type: keyword + description: Content-Transfer-Encoding of the text/[subtype]. + - name: raw + type: keyword + description: Decoded raw content of a body text type (text/[subtype] section). + - name: errors + type: object + object_type: keyword + description: Non-fatal errors while parsing MDM. + - name: external + type: group + fields: + - name: created_at + type: date + description: The created time of the message as provided by the cloud API (G Suite or Office 365) or other external source. This is typically the time the external source received the message. + - name: message_id + type: keyword + description: The message ID as provided by the cloud API (G Suite or Office 365) or other external source. + - name: route_type + type: keyword + description: whether the message was sent or received. + - name: spam + type: boolean + description: The upstream mail gateway determined the message to be spam. For cloud API providers, this will be the same as spam_folder. For other implementation methods like transport rules, this will be determined by message header values (e.g. X-SPAM) if supported. + - name: spam_folder + type: boolean + description: The message arrived in the user's spam folder. This only applies to cloud APIs (G Suite or Office 365). + - name: thread_id + type: keyword + description: The thread/conversation's unique ID as provided by the cloud API (G Suite or Office 365). + - name: headers + type: group + fields: + - name: auth_summary + type: group + fields: + - name: dmarc + type: group + fields: + - name: details + type: group + fields: + - name: action + type: keyword + description: Indicates the action taken by the spam filter based on the results of the DMARC check. For more information see https://docs.microsoft.com/en-us/microsoft-365/security/office-365-security/anti-spam-message-headers?view=o365-worldwide#authentication-results-message-header-fields. + - name: disposition + type: keyword + description: Gmail-applied policy. + - name: from + type: group + fields: + - name: domain + type: keyword + description: The fully qualified domain name (FQDN). This may not always be routable, e.g. when an email address contains a domain that is just a TLD with no SLD, e.g. foo@WIN-bar. + - name: punycode + type: keyword + description: Interpreted punycode if the domain starts with xn--. For example, if 'domain' is 'xn--ublimesecurity-4xc.com' then 'punycode' is śublimesecurity.com. + - name: root_domain + type: keyword + description: The root domain, including the TLD. + - name: sld + type: keyword + description: Second-level domain, e.g. 'windows' for the domain 'windows.net'. + - name: subdomain + type: keyword + description: Subdomain, e.g. 'drive' for the domain 'drive.google.com'. + - name: tld + type: keyword + description: The domain's top-level domain. E.g. the TLD of google.com is 'com'. + - name: valid + type: boolean + description: Whether the domain is valid. + - name: policy + type: keyword + description: Policy for the organizational domain. + - name: sub_policy + type: keyword + description: Policy for the subdomain of the organizational domain. + - name: verdict + type: keyword + description: Describes the results of the DMARC check for the message. + - name: version + type: keyword + description: DMARC version. + - name: pass + type: boolean + description: Whether the DMARC check passed. + - name: received_hop + type: long + description: The lowest hop at which the DMARC check was made. + - name: spf + type: group + fields: + - name: details + type: group + fields: + - name: client_ip + type: group + fields: + - name: ip + type: ip + description: The raw IP. + - name: description + type: keyword + description: Verbose description of the SPF verdict. + - name: designator + type: keyword + description: Email or domain of the designating body. + - name: helo + type: group + fields: + - name: domain + type: keyword + description: The fully qualified domain name (FQDN). This may not always be routable, e.g. when an email address contains a domain that is just a TLD with no SLD, e.g. foo@WIN-bar. + - name: punycode + type: keyword + description: Interpreted punycode if the domain starts with xn--. For example, if 'domain' is 'xn--ublimesecurity-4xc.com' then 'punycode' is śublimesecurity.com. + - name: root_domain + type: keyword + description: The root domain, including the TLD. + - name: sld + type: keyword + description: Second-level domain, e.g. 'windows' for the domain 'windows.net'. + - name: subdomain + type: keyword + description: Subdomain, e.g. 'drive' for the domain 'drive.google.com'. + - name: tld + type: keyword + description: The domain's top-level domain. E.g. the TLD of google.com is 'com'. + - name: valid + type: boolean + description: Whether the domain is valid. + - name: server + type: group + fields: + - name: domain + type: keyword + description: The fully qualified domain name (FQDN). This may not always be routable, e.g. when an email address contains a domain that is just a TLD with no SLD, e.g. foo@WIN-bar. + - name: punycode + type: keyword + description: Interpreted punycode if the domain starts with xn--. For example, if 'domain' is 'xn--ublimesecurity-4xc.com' then 'punycode' is śublimesecurity.com. + - name: root_domain + type: keyword + description: The root domain, including the TLD. + - name: sld + type: keyword + description: Second-level domain, e.g. 'windows' for the domain 'windows.net'. + - name: subdomain + type: keyword + description: Subdomain, e.g. 'drive' for the domain 'drive.google.com'. + - name: tld + type: keyword + description: The domain's top-level domain. E.g. the TLD of google.com is 'com'. + - name: valid + type: boolean + description: Whether the domain is valid. + - name: verdict + type: keyword + description: Verdict of the SPF. + - name: error + type: boolean + description: Whether the SPF check errored. + - name: pass + type: boolean + description: Whether the SPF check passed. + - name: received_hop + type: long + description: The lowest hop at which the SPF check was made. + - name: date + type: date + description: Date the email was sent in UTC. + - name: date_original_offset + type: keyword + description: UTC timezone offset of the sender. + - name: delivered_to + type: group + fields: + - name: domain + type: group + fields: + - name: domain + type: keyword + description: The fully qualified domain name (FQDN). This may not always be routable, e.g. when an email address contains a domain that is just a TLD with no SLD, e.g. foo@WIN-bar. + - name: punycode + type: keyword + description: Interpreted punycode if the domain starts with xn--. For example, if 'domain' is 'xn--ublimesecurity-4xc.com' then 'punycode' is śublimesecurity.com. + - name: root_domain + type: keyword + description: The root domain, including the TLD. + - name: sld + type: keyword + description: Second-level domain, e.g. 'windows' for the domain 'windows.net'. + - name: subdomain + type: keyword + description: Subdomain, e.g. 'drive' for the domain 'drive.google.com'. + - name: tld + type: keyword + description: The domain's top-level domain. E.g. the TLD of google.com is 'com'. + - name: valid + type: boolean + description: Whether the domain is valid. + - name: email + type: keyword + description: Full email address. + - name: local_part + type: keyword + description: Local-part, i.e. before the @. + - name: domains + type: group + fields: + - name: domain + type: keyword + description: The fully qualified domain name (FQDN). This may not always be routable, e.g. when an email address contains a domain that is just a TLD with no SLD, e.g. foo@WIN-bar. + - name: punycode + type: keyword + description: Interpreted punycode if the domain starts with xn--. For example, if 'domain' is 'xn--ublimesecurity-4xc.com' then 'punycode' is śublimesecurity.com. + - name: root_domain + type: keyword + description: The root domain, including the TLD. + - name: sld + type: keyword + description: Second-level domain, e.g. 'windows' for the domain 'windows.net'. + - name: subdomain + type: keyword + description: Subdomain, e.g. 'drive' for the domain 'drive.google.com'. + - name: tld + type: keyword + description: The domain's top-level domain. E.g. the TLD of google.com is 'com'. + - name: valid + type: boolean + description: Whether the domain is valid. + - name: hops + type: group + fields: + - name: authentication_results + type: group + fields: + - name: compauth + type: group + fields: + - name: reason + type: keyword + description: Reason for the verdict. + - name: verdict + type: keyword + description: Verdict of the compauth. + - name: dkim + type: keyword + description: Verdict of the Domain Keys Identified Mail check. + - name: dkim_details + type: group + fields: + - name: algorithm + type: keyword + description: Signing algorithm. + - name: body_hash + type: keyword + description: Body Hash. + - name: domain + type: keyword + description: Domain identified in the DKIM signature if any. This is the domain that's queried for the public key. + - name: headers + type: keyword + description: Header fields signed by the algorithm. + - name: instance + type: keyword + description: Instance number of this signature (if ARC). + - name: selector + type: keyword + description: Selector. + - name: signature + type: keyword + description: Signature of headers and body. + - name: type + type: keyword + description: The type of signature, derived from the field name. + - name: version + type: keyword + description: Version. + - name: dmarc + type: keyword + description: Verdict of the Domain-based Message Authentication, Reporting & Conformance check. + - name: dmarc_details + type: group + fields: + - name: action + type: keyword + description: Indicates the action taken by the spam filter based on the results of the DMARC check. For more information see https://docs.microsoft.com/en-us/microsoft-365/security/office-365-security/anti-spam-message-headers?view=o365-worldwide#authentication-results-message-header-fields. + - name: disposition + type: keyword + description: Gmail-applied policy. + - name: from + type: group + fields: + - name: domain + type: keyword + description: The fully qualified domain name (FQDN). This may not always be routable, e.g. when an email address contains a domain that is just a TLD with no SLD, e.g. foo@WIN-bar. + - name: punycode + type: keyword + description: Interpreted punycode if the domain starts with xn--. For example, if 'domain' is 'xn--ublimesecurity-4xc.com' then 'punycode' is śublimesecurity.com. + - name: root_domain + type: keyword + description: The root domain, including the TLD. + - name: sld + type: keyword + description: Second-level domain, e.g. 'windows' for the domain 'windows.net'. + - name: subdomain + type: keyword + description: Subdomain, e.g. 'drive' for the domain 'drive.google.com'. + - name: tld + type: keyword + description: The domain's top-level domain. E.g. the TLD of google.com is 'com'. + - name: valid + type: boolean + description: Whether the domain is valid. + - name: policy + type: keyword + description: Policy for the organizational domain. + - name: sub_policy + type: keyword + description: Policy for the subdomain of the organizational domain. + - name: verdict + type: keyword + description: Describes the results of the DMARC check for the message. + - name: version + type: keyword + description: DMARC version. + - name: instance + type: keyword + description: Instance number of this auth result (if ARC). + - name: server + type: group + fields: + - name: domain + type: keyword + description: The fully qualified domain name (FQDN). This may not always be routable, e.g. when an email address contains a domain that is just a TLD with no SLD, e.g. foo@WIN-bar. + - name: punycode + type: keyword + description: Interpreted punycode if the domain starts with xn--. For example, if 'domain' is 'xn--ublimesecurity-4xc.com' then 'punycode' is śublimesecurity.com. + - name: root_domain + type: keyword + description: The root domain, including the TLD. + - name: sld + type: keyword + description: Second-level domain, e.g. 'windows' for the domain 'windows.net'. + - name: subdomain + type: keyword + description: Subdomain, e.g. 'drive' for the domain 'drive.google.com'. + - name: tld + type: keyword + description: The domain's top-level domain. E.g. the TLD of google.com is 'com'. + - name: valid + type: boolean + description: Whether the domain is valid. + - name: spf + type: keyword + description: Verdict of the Sender Policy Framework. + - name: spf_details + type: group + fields: + - name: client_ip + type: group + fields: + - name: ip + type: ip + description: The raw IP. + - name: description + type: keyword + description: Verbose description of the SPF verdict. + - name: designator + type: keyword + description: Email or domain of the designating body. + - name: helo + type: group + fields: + - name: domain + type: keyword + description: The fully qualified domain name (FQDN). This may not always be routable, e.g. when an email address contains a domain that is just a TLD with no SLD, e.g. foo@WIN-bar. + - name: punycode + type: keyword + description: Interpreted punycode if the domain starts with xn--. For example, if 'domain' is 'xn--ublimesecurity-4xc.com' then 'punycode' is śublimesecurity.com. + - name: root_domain + type: keyword + description: The root domain, including the TLD. + - name: sld + type: keyword + description: Second-level domain, e.g. 'windows' for the domain 'windows.net'. + - name: subdomain + type: keyword + description: Subdomain, e.g. 'drive' for the domain 'drive.google.com'. + - name: tld + type: keyword + description: The domain's top-level domain. E.g. the TLD of google.com is 'com'. + - name: valid + type: boolean + description: Whether the domain is valid. + - name: server + type: group + fields: + - name: domain + type: keyword + description: The fully qualified domain name (FQDN). This may not always be routable, e.g. when an email address contains a domain that is just a TLD with no SLD, e.g. foo@WIN-bar. + - name: punycode + type: keyword + description: Interpreted punycode if the domain starts with xn--. For example, if 'domain' is 'xn--ublimesecurity-4xc.com' then 'punycode' is śublimesecurity.com. + - name: root_domain + type: keyword + description: The root domain, including the TLD. + - name: sld + type: keyword + description: Second-level domain, e.g. 'windows' for the domain 'windows.net'. + - name: subdomain + type: keyword + description: Subdomain, e.g. 'drive' for the domain 'drive.google.com'. + - name: tld + type: keyword + description: The domain's top-level domain. E.g. the TLD of google.com is 'com'. + - name: valid + type: boolean + description: Whether the domain is valid. + - name: verdict + type: keyword + description: Verdict of the SPF. + - name: type + type: keyword + description: The type of authentication result, derived from the field name. + - name: fields + type: object + object_type: keyword + - name: index + type: long + description: Index indicates the order in which a hop occurred from sender to recipient. + - name: received + type: group + fields: + - name: additional + type: group + fields: + - name: raw + type: keyword + description: The raw string for remaining additional clauses, such as transport information. + - name: id + type: group + fields: + - name: raw + type: keyword + description: The raw string of 'id' section. + - name: link + type: group + fields: + - name: raw + type: keyword + description: The raw string of 'via' section. + - name: mailbox + type: group + fields: + - name: raw + type: keyword + description: The raw string of 'for' section. + - name: protocol + type: group + fields: + - name: raw + type: keyword + description: The raw string of 'with' section. + - name: server + type: group + fields: + - name: raw + type: keyword + description: The raw string of 'by' section. + - name: source + type: group + fields: + - name: raw + type: keyword + description: The raw string of 'from' section. + - name: time + type: date + description: Time parsed from the Received header. + - name: zone_offset + type: keyword + description: Timezone offset parsed from the Received header. + - name: received_spf + type: group + fields: + - name: client_ip + type: group + fields: + - name: ip + type: ip + description: The raw IP. + - name: description + type: keyword + description: Verbose description of the SPF verdict. + - name: designator + type: keyword + description: Email or domain of the designating body. + - name: helo + type: group + fields: + - name: domain + type: keyword + description: The fully qualified domain name (FQDN). This may not always be routable, e.g. when an email address contains a domain that is just a TLD with no SLD, e.g. foo@WIN-bar. + - name: punycode + type: keyword + description: Interpreted punycode if the domain starts with xn--. For example, if 'domain' is 'xn--ublimesecurity-4xc.com' then 'punycode' is śublimesecurity.com. + - name: root_domain + type: keyword + description: The root domain, including the TLD. + - name: sld + type: keyword + description: Second-level domain, e.g. 'windows' for the domain 'windows.net'. + - name: subdomain + type: keyword + description: Subdomain, e.g. 'drive' for the domain 'drive.google.com'. + - name: tld + type: keyword + description: The domain's top-level domain. E.g. the TLD of google.com is 'com'. + - name: valid + type: boolean + description: Whether the domain is valid. + - name: server + type: group + fields: + - name: domain + type: keyword + description: The fully qualified domain name (FQDN). This may not always be routable, e.g. when an email address contains a domain that is just a TLD with no SLD, e.g. foo@WIN-bar. + - name: punycode + type: keyword + description: Interpreted punycode if the domain starts with xn--. For example, if 'domain' is 'xn--ublimesecurity-4xc.com' then 'punycode' is śublimesecurity.com. + - name: root_domain + type: keyword + description: The root domain, including the TLD. + - name: sld + type: keyword + description: Second-level domain, e.g. 'windows' for the domain 'windows.net'. + - name: subdomain + type: keyword + description: Subdomain, e.g. 'drive' for the domain 'drive.google.com'. + - name: tld + type: keyword + description: The domain's top-level domain. E.g. the TLD of google.com is 'com'. + - name: valid + type: boolean + description: Whether the domain is valid. + - name: verdict + type: keyword + description: Verdict of the SPF. + - name: signature + type: group + fields: + - name: algorithm + type: keyword + description: Signing algorithm. + - name: body_hash + type: keyword + description: Body Hash. + - name: domain + type: keyword + description: Domain identified in the DKIM signature if any. This is the domain that's queried for the public key. + - name: headers + type: keyword + description: Header fields signed by the algorithm. + - name: instance + type: keyword + description: Instance number of this signature (if ARC). + - name: selector + type: keyword + description: Selector. + - name: signature + type: keyword + description: Signature of headers and body. + - name: type + type: keyword + description: The type of signature, derived from the field name. + - name: version + type: keyword + description: Version. + - name: in_reply_to + type: keyword + description: In-Reply-To header value which identifies its parent message if exists. + - name: ips + type: group + fields: + - name: ip + type: keyword + description: The raw IP. + - name: mailer + type: keyword + description: X-Mailer or User-Agent extracted from headers. + - name: message_id + type: keyword + description: Message-ID extracted from the header. + - name: references + type: keyword + description: The Message-IDs of the other messages within this chain. + - name: reply_to + type: group + fields: + - name: display_name + type: keyword + description: Display name. + - name: email + type: group + fields: + - name: domain + type: group + fields: + - name: domain + type: keyword + description: The fully qualified domain name (FQDN). This may not always be routable, e.g. when an email address contains a domain that is just a TLD with no SLD, e.g. foo@WIN-bar. + - name: punycode + type: keyword + description: Interpreted punycode if the domain starts with xn--. For example, if 'domain' is 'xn--ublimesecurity-4xc.com' then 'punycode' is śublimesecurity.com. + - name: root_domain + type: keyword + description: The root domain, including the TLD. + - name: sld + type: keyword + description: Second-level domain, e.g. 'windows' for the domain 'windows.net'. + - name: subdomain + type: keyword + description: Subdomain, e.g. 'drive' for the domain 'drive.google.com'. + - name: tld + type: keyword + description: The domain's top-level domain. E.g. the TLD of google.com is 'com'. + - name: valid + type: boolean + description: Whether the domain is valid. + - name: local_part + type: keyword + description: Local-part, i.e. before the @. + - name: value + type: keyword + description: Full email address. + - name: return_path + type: group + fields: + - name: domain + type: group + fields: + - name: domain + type: keyword + description: The fully qualified domain name (FQDN). This may not always be routable, e.g. when an email address contains a domain that is just a TLD with no SLD, e.g. foo@WIN-bar. + - name: punycode + type: keyword + description: Interpreted punycode if the domain starts with xn--. For example, if 'domain' is 'xn--ublimesecurity-4xc.com' then 'punycode' is śublimesecurity.com. + - name: root_domain + type: keyword + description: The root domain, including the TLD. + - name: sld + type: keyword + description: Second-level domain, e.g. 'windows' for the domain 'windows.net'. + - name: subdomain + type: keyword + description: Subdomain, e.g. 'drive' for the domain 'drive.google.com'. + - name: tld + type: keyword + description: The domain's top-level domain. E.g. the TLD of google.com is 'com'. + - name: valid + type: boolean + description: Whether the domain is valid. + - name: email + type: keyword + description: Full email address. + - name: local_part + type: keyword + description: Local-part, i.e. before the @. + - name: x_authenticated_domain + type: group + fields: + - name: domain + type: keyword + description: The fully qualified domain name (FQDN). This may not always be routable, e.g. when an email address contains a domain that is just a TLD with no SLD, e.g. foo@WIN-bar. + - name: punycode + type: keyword + description: Interpreted punycode if the domain starts with xn--. For example, if 'domain' is 'xn--ublimesecurity-4xc.com' then 'punycode' is śublimesecurity.com. + - name: root_domain + type: keyword + description: The root domain, including the TLD. + - name: sld + type: keyword + description: Second-level domain, e.g. 'windows' for the domain 'windows.net'. + - name: subdomain + type: keyword + description: Subdomain, e.g. 'drive' for the domain 'drive.google.com'. + - name: tld + type: keyword + description: The domain's top-level domain. E.g. the TLD of google.com is 'com'. + - name: valid + type: boolean + description: Whether the domain is valid. + - name: x_authenticated_sender + type: group + fields: + - name: domain + type: group + fields: + - name: domain + type: keyword + description: The fully qualified domain name (FQDN). This may not always be routable, e.g. when an email address contains a domain that is just a TLD with no SLD, e.g. foo@WIN-bar. + - name: punycode + type: keyword + description: Interpreted punycode if the domain starts with xn--. For example, if 'domain' is 'xn--ublimesecurity-4xc.com' then 'punycode' is śublimesecurity.com. + - name: root_domain + type: keyword + description: The root domain, including the TLD. + - name: sld + type: keyword + description: Second-level domain, e.g. 'windows' for the domain 'windows.net'. + - name: subdomain + type: keyword + description: Subdomain, e.g. 'drive' for the domain 'drive.google.com'. + - name: tld + type: keyword + description: The domain's top-level domain. E.g. the TLD of google.com is 'com'. + - name: valid + type: boolean + description: Whether the domain is valid. + - name: email + type: keyword + description: Full email address. + - name: local_part + type: keyword + description: Local-part, i.e. before the @. + - name: x_client_ip + type: group + fields: + - name: ip + type: ip + description: The raw IP. + - name: x_originating_ip + type: group + fields: + - name: ip + type: ip + description: The raw IP. + - name: x_secure_server_account + type: keyword + description: X-SecureServer-Acct header, which represents a unique identifier associated with the sender's email account on a secure server and can be used to trace the email back to a specific account or user. + - name: x_sender + type: group + fields: + - name: domain + type: group + fields: + - name: domain + type: keyword + description: The fully qualified domain name (FQDN). This may not always be routable, e.g. when an email address contains a domain that is just a TLD with no SLD, e.g. foo@WIN-bar. + - name: punycode + type: keyword + description: Interpreted punycode if the domain starts with xn--. For example, if 'domain' is 'xn--ublimesecurity-4xc.com' then 'punycode' is śublimesecurity.com. + - name: root_domain + type: keyword + description: The root domain, including the TLD. + - name: sld + type: keyword + description: Second-level domain, e.g. 'windows' for the domain 'windows.net'. + - name: subdomain + type: keyword + description: Subdomain, e.g. 'drive' for the domain 'drive.google.com'. + - name: tld + type: keyword + description: The domain's top-level domain. E.g. the TLD of google.com is 'com'. + - name: valid + type: boolean + description: Whether the domain is valid. + - name: email + type: keyword + description: Full email address. + - name: local_part + type: keyword + description: Local-part, i.e. before the @. + - name: mailbox + type: group + fields: + - name: display_name + type: keyword + description: Display name. + - name: email + type: group + fields: + - name: domain + type: group + fields: + - name: domain + type: keyword + description: The fully qualified domain name (FQDN). This may not always be routable, e.g. when an email address contains a domain that is just a TLD with no SLD, e.g. foo@WIN-bar. + - name: punycode + type: keyword + description: Interpreted punycode if the domain starts with xn--. For example, if 'domain' is 'xn--ublimesecurity-4xc.com' then 'punycode' is śublimesecurity.com. + - name: root_domain + type: keyword + description: The root domain, including the TLD. + - name: sld + type: keyword + description: Second-level domain, e.g. 'windows' for the domain 'windows.net'. + - name: subdomain + type: keyword + description: Subdomain, e.g. 'drive' for the domain 'drive.google.com'. + - name: tld + type: keyword + description: The domain's top-level domain. E.g. the TLD of google.com is 'com'. + - name: valid + type: boolean + description: Whether the domain is valid. + - name: local_part + type: keyword + description: Local-part, i.e. before the @. + - name: value + type: keyword + description: Full email address. + - name: meta + type: group + fields: + - name: canonical_id + type: keyword + description: A deterministic ID, generated from metadata such as Attachments, Body, Subject, Sender and is used to group similar messages/campaigns together. + - name: created_at + type: date + description: Creation time of the data model. + - name: effective_at + type: date + description: Effective time of the data model, used for evaluation against lists and historical functions such as sender profiles or whois. + - name: id + type: keyword + description: Message ID. + - name: recipients + type: group + fields: + - name: bcc + type: group + fields: + - name: display_name + type: keyword + description: Display name. + - name: email + type: group + fields: + - name: domain + type: group + fields: + - name: domain + type: keyword + description: The fully qualified domain name (FQDN). This may not always be routable, e.g. when an email address contains a domain that is just a TLD with no SLD, e.g. foo@WIN-bar. + - name: punycode + type: keyword + description: Interpreted punycode if the domain starts with xn--. For example, if 'domain' is 'xn--ublimesecurity-4xc.com' then 'punycode' is śublimesecurity.com. + - name: root_domain + type: keyword + description: The root domain, including the TLD. + - name: sld + type: keyword + description: Second-level domain, e.g. 'windows' for the domain 'windows.net'. + - name: subdomain + type: keyword + description: Subdomain, e.g. 'drive' for the domain 'drive.google.com'. + - name: tld + type: keyword + description: The domain's top-level domain. E.g. the TLD of google.com is 'com'. + - name: valid + type: boolean + description: Whether the domain is valid. + - name: local_part + type: keyword + description: Local-part, i.e. before the @. + - name: value + type: keyword + description: Full email address. + - name: cc + type: group + fields: + - name: display_name + type: keyword + description: Display name. + - name: email + type: group + fields: + - name: domain + type: group + fields: + - name: domain + type: keyword + description: The fully qualified domain name (FQDN). This may not always be routable, e.g. when an email address contains a domain that is just a TLD with no SLD, e.g. foo@WIN-bar. + - name: punycode + type: keyword + description: Interpreted punycode if the domain starts with xn--. For example, if 'domain' is 'xn--ublimesecurity-4xc.com' then 'punycode' is śublimesecurity.com. + - name: root_domain + type: keyword + description: The root domain, including the TLD. + - name: sld + type: keyword + description: Second-level domain, e.g. 'windows' for the domain 'windows.net'. + - name: subdomain + type: keyword + description: Subdomain, e.g. 'drive' for the domain 'drive.google.com'. + - name: tld + type: keyword + description: The domain's top-level domain. E.g. the TLD of google.com is 'com'. + - name: valid + type: boolean + description: Whether the domain is valid. + - name: local_part + type: keyword + description: Local-part, i.e. before the @. + - name: value + type: keyword + description: Full email address. + - name: to + type: group + fields: + - name: display_name + type: keyword + description: Display name. + - name: email + type: group + fields: + - name: domain + type: group + fields: + - name: domain + type: keyword + description: The fully qualified domain name (FQDN). This may not always be routable, e.g. when an email address contains a domain that is just a TLD with no SLD, e.g. foo@WIN-bar. + - name: punycode + type: keyword + description: Interpreted punycode if the domain starts with xn--. For example, if 'domain' is 'xn--ublimesecurity-4xc.com' then 'punycode' is śublimesecurity.com. + - name: root_domain + type: keyword + description: The root domain, including the TLD. + - name: sld + type: keyword + description: Second-level domain, e.g. 'windows' for the domain 'windows.net'. + - name: subdomain + type: keyword + description: Subdomain, e.g. 'drive' for the domain 'drive.google.com'. + - name: tld + type: keyword + description: The domain's top-level domain. E.g. the TLD of google.com is 'com'. + - name: valid + type: boolean + description: Whether the domain is valid. + - name: local_part + type: keyword + description: Local-part, i.e. before the @. + - name: value + type: keyword + description: Full email address. + - name: sender + type: group + fields: + - name: display_name + type: keyword + description: Display name. + - name: email + type: group + fields: + - name: domain + type: group + fields: + - name: domain + type: keyword + description: The fully qualified domain name (FQDN). This may not always be routable, e.g. when an email address contains a domain that is just a TLD with no SLD, e.g. foo@WIN-bar. + - name: punycode + type: keyword + description: Interpreted punycode if the domain starts with xn--. For example, if 'domain' is 'xn--ublimesecurity-4xc.com' then 'punycode' is śublimesecurity.com. + - name: root_domain + type: keyword + description: The root domain, including the TLD. + - name: sld + type: keyword + description: Second-level domain, e.g. 'windows' for the domain 'windows.net'. + - name: subdomain + type: keyword + description: Subdomain, e.g. 'drive' for the domain 'drive.google.com'. + - name: tld + type: keyword + description: The domain's top-level domain. E.g. the TLD of google.com is 'com'. + - name: valid + type: boolean + description: Whether the domain is valid. + - name: local_part + type: keyword + description: Local-part, i.e. before the @. + - name: value + type: keyword + description: Full email address. + - name: subject + type: group + fields: + - name: subject + type: keyword + description: Subject of the email. + - name: type + type: group + fields: + - name: inbound + type: boolean + description: Message was sent from someone outside your organization, to at least one recipient inside your organization. + - name: internal + type: boolean + description: Message was sent from someone inside your organization, to at least one recipient inside your organization. Messages must be authenticated by either SPF or DKIM to be treated as internal. + - name: outbound + type: boolean + description: Message was sent from someone inside your organization, to at least one recipient outside your organization. diff --git a/packages/sublime_security/data_stream/email_message/manifest.yml b/packages/sublime_security/data_stream/email_message/manifest.yml new file mode 100644 index 00000000000..232f07c86e2 --- /dev/null +++ b/packages/sublime_security/data_stream/email_message/manifest.yml @@ -0,0 +1,117 @@ +title: Sublime Security Email Message logs +type: logs +streams: + - input: aws-s3 + template_path: aws-s3.yml.hbs + title: Sublime Security Email Message logs via AWS S3 or SQS + description: Collecting Sublime Security Email Message logs via AWS S3 or SQS input. + enabled: false + vars: + - name: bucket_arn + type: text + title: '[S3] Bucket ARN' + multi: false + required: false + show_user: true + description: ARN of the AWS S3 bucket that will be polled for list operation. It is a required parameter for collecting logs via the AWS S3. + - name: bucket_list_prefix + type: text + title: '[S3] Bucket Prefix' + multi: false + required: false + show_user: true + description: Prefix to apply for the list request to the S3 bucket. + - name: interval + type: text + title: '[S3] Interval' + multi: false + required: false + show_user: true + default: 120s + description: Listing of the S3 bucket will be polled according to the time interval defined by bucket_list_interval config. Default value is 120 secs. Supported units for this parameter are h/m/s. + - name: number_of_workers + type: integer + title: '[S3] Number of Workers' + multi: false + required: false + show_user: true + default: 5 + description: Number of workers that will process the S3 objects listed. + - name: queue_url + type: text + title: '[SQS] Queue URL' + multi: false + required: false + show_user: true + description: URL of the AWS SQS queue that messages will be received from. It is a required parameter for collecting logs via the AWS SQS. + - name: visibility_timeout + type: text + title: '[SQS] Visibility Timeout' + multi: false + required: false + show_user: true + default: 300s + description: The duration that the received messages are hidden from subsequent retrieve requests after being retrieved by a ReceiveMessage request. The maximum is 12 hours. Supported units for this parameter are h/m/s. + - name: api_timeout + type: text + title: '[SQS] API Timeout' + multi: false + required: false + show_user: true + default: 120s + description: The maximum duration of AWS API can take. The maximum is half of the visibility timeout value. Supported units for this parameter are h/m/s. + - name: max_number_of_messages + type: integer + title: '[SQS] Maximum Concurrent SQS Messages' + required: false + show_user: true + default: 5 + description: The maximum number of SQS messages that can be inflight at any time. + - name: file_selectors + type: yaml + title: '[SQS] File Selectors' + multi: false + required: false + show_user: false + description: >- + If the SQS queue will have events that correspond to files that this integration shouldn’t process, file_selectors can be used to limit the files that are downloaded. This is a list of selectors which are made up of regex and expand_event_list_from_field options. The regex should match the S3 object key in the SQS message, and the optional expand_event_list_from_field is the same as the global setting. If file_selectors is given, then any global expand_event_list_from_field value is ignored in favor of the ones specified in the file_selectors. Regexes use [RE2 syntax](https://pkg.go.dev/regexp/syntax). Files that do not match one of the regexes will not be processed. + - name: external_id + type: text + title: External ID + multi: false + required: false + show_user: false + description: External ID to use when assuming a role in another account. + - name: tags + type: text + title: Tags + multi: true + required: true + show_user: false + default: + - forwarded + - sublime_security-email_message + - name: preserve_original_event + required: false + show_user: true + title: Preserve original event + description: Preserves a raw copy of the original event, added to the field `event.original`. + type: bool + multi: false + default: false + - name: preserve_duplicate_custom_fields + required: true + show_user: false + title: Preserve duplicate custom fields + description: Preserve sublime_security.email_message fields that were mapped to Elastic Common Schema (ECS) fields. + type: bool + multi: false + default: false + - name: processors + type: yaml + title: Processors + multi: false + required: false + show_user: false + description: >- + Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See [Processors](https://www.elastic.co/guide/en/beats/filebeat/current/filtering-and-enhancing-data.html) for details. diff --git a/packages/sublime_security/data_stream/email_message/sample_event.json b/packages/sublime_security/data_stream/email_message/sample_event.json new file mode 100644 index 00000000000..bbb0f3711e2 --- /dev/null +++ b/packages/sublime_security/data_stream/email_message/sample_event.json @@ -0,0 +1,762 @@ +{ + "@timestamp": "2024-08-02T07:40:25.135Z", + "agent": { + "ephemeral_id": "832ebf28-565e-4f38-a67e-ee5ea9f51e89", + "id": "5f3fcbb9-1a97-4ff3-857f-167af6664464", + "name": "docker-fleet-agent", + "type": "filebeat", + "version": "8.13.0" + }, + "aws": { + "s3": { + "bucket": { + "arn": "arn:aws:s3:::elastic-package-sublime-security-bucket-33881", + "name": "elastic-package-sublime-security-bucket-33881" + }, + "object": { + "key": "email-message.log" + } + } + }, + "cloud": { + "region": "us-east-1" + }, + "data_stream": { + "dataset": "sublime_security.email_message", + "namespace": "57022", + "type": "logs" + }, + "destination": { + "domain": "example.com", + "subdomain": "example", + "top_level_domain": "com" + }, + "ecs": { + "version": "8.11.0" + }, + "elastic_agent": { + "id": "5f3fcbb9-1a97-4ff3-857f-167af6664464", + "snapshot": false, + "version": "8.13.0" + }, + "email": { + "attachments": [ + { + "file": { + "extension": "pdf", + "hash": { + "md5": "1a2b3c", + "sha1": "4d5e6f", + "sha256": "7g8h9i" + }, + "mime_type": "application/pdf", + "name": "sample_document.pdf", + "size": 102400 + } + }, + { + "file": { + "extension": "jpg", + "hash": { + "md5": "7h8i9j", + "sha1": "1k2l3m", + "sha256": "4n5o6p" + }, + "mime_type": "image/jpeg", + "name": "image_photo.jpg", + "size": 204800 + } + }, + { + "file": { + "extension": "txt", + "hash": { + "md5": "1x2y3z", + "sha1": "4a5b6c", + "sha256": "7d8e9f" + }, + "mime_type": "text/plain", + "name": "notes.txt", + "size": 5120 + } + } + ], + "bcc": { + "address": [ + "john.doe@example.com" + ] + }, + "cc": { + "address": [ + "jane.smith@example.org" + ] + }, + "direction": "outbound", + "from": { + "address": [ + "testing@sublimesecurity.com" + ] + }, + "message_id": "2fe271830bbad5fe3a70abbe7a8c0bfe7refe3ffe", + "origination_timestamp": "2024-08-02T07:40:25.135Z", + "reply_to": { + "address": [ + "user@example.com" + ] + }, + "subject": "Sublime-Security-Standard-Test-String", + "to": { + "address": [ + "user@example.com" + ] + }, + "x_mailer": "MyCustomMailer" + }, + "event": { + "agent_id_status": "verified", + "category": [ + "email" + ], + "dataset": "sublime_security.email_message", + "id": "01911208-633c-7f03-b303-e594d92cf818", + "ingested": "2024-08-28T10:59:37Z", + "kind": "event", + "original": "{\"body\":{\"plain\":{\"raw\":\"Sublime Security test message.\\n\",\"charset\":\"utf-8\",\"content_transfer_encoding\":\"base64\"},\"current_thread\":{\"text\":\"Sublime Security test message.\"},\"html\":{\"charset\":\"utf-8\",\"content_transfer_encoding\":\"base64\",\"display_text\":\"Sublime Security test message.\",\"raw\":\"

Sublime Security test message.

\",\"inner_text\":\"

Sublime Security test message.

\"},\"ips\":[{\"ip\":\"1.128.0.0\"}],\"links\":[{\"display_text\":\"Click here!\",\"mismatched\":true,\"display_url\":{\"fragment\":\"search\",\"password\":\"pass123\",\"path\":\"/test\",\"port\":80,\"query_params\":\"q=elasticsearch\",\"rewrite\":{\"encoders\":[\"base64\"],\"original\":\"demo\"},\"scheme\":\"https\",\"url\":\"https://example.com/test?q=elasticsearch#search\",\"username\":\"test\",\"domain\":{\"domain\":\"example.com\",\"punycode\":\"demo\",\"root_domain\":\"example.com\",\"subdomain\":\"example\",\"tld\":\"com\",\"valid\":true,\"sld\":\"example\"}}},{\"href_url\":{\"fragment\":\"search\",\"password\":\"pass123\",\"path\":\"/test\",\"port\":80,\"query_params\":\"q=elasticsearch\",\"rewrite\":{\"encoders\":[\"base64\"],\"original\":\"demo\"},\"scheme\":\"https\",\"url\":\"https://example.com/test?q=elasticsearch#search\",\"username\":\"test\",\"domain\":{\"domain\":\"example.com\",\"punycode\":\"demo\",\"root_domain\":\"example.com\",\"subdomain\":\"example\",\"tld\":\"com\",\"valid\":true,\"sld\":\"example\"}}}]},\"external\":{\"created_at\":\"2024-08-02T07:40:25.135939305Z\",\"message_id\":\"2fe271830bbad5fe3a70abbe7a8c0bfe7refe3ffe\",\"route_type\":\"sent\",\"spam\":false,\"spam_folder\":true,\"thread_id\":\"sample_data\"},\"attachments\":[{\"content_id\":\"abc123\",\"content_transfer_encoding\":\"base64\",\"content_type\":\"application/pdf\",\"file_extension\":\".pdf\",\"file_name\":\"sample_document.pdf\",\"file_type\":\"document\",\"md5\":\"1a2b3c\",\"raw\":\"JVBERi0xLjMKJcfs4AAQSkZjRgABAQE\",\"sha1\":\"4d5e6f\",\"sha256\":\"7g8h9i\",\"size\":102400},{\"content_id\":\"xyz456\",\"content_transfer_encoding\":\"7bit\",\"content_type\":\"image/jpeg\",\"file_extension\":\".jpg\",\"file_name\":\"image_photo.jpg\",\"file_type\":\"image\",\"md5\":\"7h8i9j\",\"raw\":\"/9j/4AAQSkZJRgABAQEJVBERi0xLjMKJd\",\"sha1\":\"1k2l3m\",\"sha256\":\"4n5o6p\",\"size\":204800},{\"content_id\":\"efg789\",\"content_transfer_encoding\":\"quoted-printable\",\"content_type\":\"text/plain\",\"file_extension\":\".txt\",\"file_name\":\"notes.txt\",\"file_type\":\"text\",\"md5\":\"1x2y3z\",\"raw\":\"SGVsbG8gdVsbG8gd29yb29ybGQhVsbG8gd29yb\",\"sha1\":\"4a5b6c\",\"sha256\":\"7d8e9f\",\"size\":5120}],\"headers\":{\"x_authenticated_domain\":{\"domain\":\"example.com\",\"punycode\":\"xn--example-d4a.com\",\"root_domain\":\"example.com\",\"sld\":\"example\",\"subdomain\":\"sub\",\"tld\":\"com\",\"valid\":true},\"x_authenticated_sender\":{\"domain\":{\"domain\":\"example.com\",\"punycode\":\"example.com\",\"root_domain\":\"example.com\",\"sld\":\"example\",\"subdomain\":\"sub\",\"tld\":\"com\",\"valid\":true},\"email\":\"user@example.com\",\"local_part\":\"user\"},\"x_client_ip\":{\"ip\":\"1.128.0.0\"},\"x_originating_ip\":{\"ip\":\"1.128.0.0\"},\"x_secure_server_account\":\"account_value\",\"x_sender\":{\"domain\":{\"domain\":\"example.com\",\"punycode\":\"example.com\",\"root_domain\":\"example.com\",\"sld\":\"example\",\"subdomain\":\"sub\",\"tld\":\"com\",\"valid\":true},\"email\":\"user@example.com\",\"local_part\":\"user\"},\"return_path\":{\"domain\":{\"domain\":\"example.com\",\"punycode\":\"xn--example-d4a.com\",\"root_domain\":\"example\",\"sld\":\"example\",\"subdomain\":\"sub\",\"tld\":\"com\",\"valid\":true},\"email\":\"user@example.com\",\"local_part\":\"user\"},\"references\":[\"test1\",\"test2\"],\"auth_summary\":{\"dmarc\":{\"details\":{\"action\":\"quarantine\",\"disposition\":\"quarantine\",\"from\":{\"domain\":\"example.com\",\"punycode\":\"example.com\",\"root_domain\":\"example.com\",\"sld\":\"example\",\"subdomain\":\"example\",\"tld\":\"com\",\"valid\":true},\"policy\":\"reject\",\"sub_policy\":\"none\",\"verdict\":\"pass\",\"version\":\"1.0\"},\"pass\":true,\"received_hop\":1},\"spf\":{\"details\":{\"client_ip\":{\"ip\":\"1.128.0.0\"},\"description\":\"SPF record found\",\"designator\":\"pass\",\"helo\":{\"domain\":\"example.com\",\"punycode\":\"example.com\",\"root_domain\":\"example.com\",\"sld\":\"example\",\"subdomain\":\"example\",\"tld\":\"com\",\"valid\":true},\"server\":{\"domain\":\"mail.example.com\",\"punycode\":\"mail.example.com\",\"root_domain\":\"example.com\",\"sld\":\"example\",\"subdomain\":\"mail\",\"tld\":\"com\",\"valid\":true},\"verdict\":\"pass\"},\"error\":\"true\",\"pass\":true,\"received_hop\":2}},\"date\":\"2019-10-21T18:23:24Z\",\"date_original_offset\":\"-4\",\"hops\":[{\"index\":0,\"fields\":[{\"name\":\"To\",\"value\":\"user@example.com\",\"position\":0},{\"name\":\"Subject\",\"value\":\"Sublime-Security-Standard-Test-String\",\"position\":1},{\"name\":\"Date\",\"value\":\"Mon, 21 Oct 2019 14:23:24 -0400\",\"position\":2},{\"name\":\"From\",\"value\":\"Sublime Security Test \",\"position\":3}],\"authentication_results\":{\"compauth\":{\"verdict\":\"pass\",\"reason\":\"reason_value\"},\"dkim\":\"pass\",\"dkim_details\":{\"algorithm\":\"rsa-sha256\",\"body_hash\":\"abcdefg\",\"domain\":\"example.com\",\"headers\":\"from, to, subject\",\"instance\":\"example.com\",\"selector\":\"abcdefg\",\"signature\":\"abcdefg\",\"type\":\"dkim\",\"version\":\"1.0\"},\"dmarc\":\"pass\",\"dmarc_details\":{\"action\":\"quarantine\",\"disposition\":\"quarantine\",\"from\":{\"domain\":\"example.com\",\"punycode\":\"example.com\",\"root_domain\":\"example.com\",\"sld\":\"example\",\"subdomain\":\"example\",\"tld\":\"com\",\"valid\":true},\"policy\":\"reject\",\"sub_policy\":\"none\",\"verdict\":\"pass\",\"version\":\"1.0\"},\"instance\":\"example.com\",\"server\":{\"domain\":\"mail.example.com\",\"punycode\":\"mail.example.com\",\"root_domain\":\"example.com\",\"sld\":\"example\",\"subdomain\":\"mail\",\"tld\":\"com\",\"valid\":true},\"spf\":\"pass\",\"spf_details\":{\"client_ip\":{\"ip\":\"1.128.0.0\"},\"description\":\"SPF record found\",\"designator\":\"pass\",\"helo\":{\"domain\":\"example.com\",\"punycode\":\"example.com\",\"root_domain\":\"example.com\",\"sld\":\"example\",\"subdomain\":\"example\",\"tld\":\"com\",\"valid\":true},\"server\":{\"domain\":\"mail.example.com\",\"root_domain\":\"example.com\",\"sld\":\"example\",\"subdomain\":\"mail\",\"tld\":\"com\",\"valid\":true},\"verdict\":\"pass\"},\"type\":\"spf\"},\"received\":{\"additional\":{\"raw\":\"Authentication successful\"},\"id\":{\"raw\":\"msg-12345\"},\"link\":{\"raw\":\"https://mail.example.com/message/12345\"},\"mailbox\":{\"raw\":\"user@example.com\"},\"protocol\":{\"raw\":\"IMAP\"},\"server\":{\"raw\":\"imap.example.com\"},\"source\":{\"raw\":\"81.2.69.144\"},\"time\":\"2019-10-21T18:23:24Z\",\"zone_offset\":\"+00:00\"},\"received_spf\":{\"client_ip\":{\"ip\":\"1.128.0.0\"},\"description\":\"SPF record found\",\"designator\":\"pass\",\"helo\":{\"domain\":\"example.com\",\"punycode\":\"example.com\",\"root_domain\":\"example.com\",\"sld\":\"example\",\"subdomain\":\"example\",\"tld\":\"com\",\"valid\":true},\"server\":{\"domain\":\"mail.example.com\",\"punycode\":\"mail.example.com\",\"root_domain\":\"example.com\",\"sld\":\"example\",\"subdomain\":\"mail\",\"tld\":\"com\",\"valid\":true},\"verdict\":\"pass\"},\"signature\":{\"algorithm\":\"rsa-sha256\",\"body_hash\":\"b9c4a3f9d93d9a38bdf8c47a8f2d2c79ec1d8b1f\",\"domain\":\"example.com\",\"headers\":\"from:to:subject:date\",\"instance\":\"123456\",\"selector\":\"default\",\"signature\":\"d2abf9d6c8f4b8d68d8f3f7b6f9d3b8e6a8c2b3a9f4b8d7b9d3b6a8f9c3b4e5f\",\"type\":\"spf\",\"version\":\"1\"}}],\"in_reply_to\":\"in_reply_to_value\",\"delivered_to\":{\"domain\":{\"domain\":\"example.com\",\"subdomain\":\"example\",\"tld\":\"com\",\"email\":\"testing@sublimesecurity.com\",\"punycode\":\"example.com\",\"root_domain\":\"example.com\",\"sld\":\"example\",\"valid\":true},\"email\":\"testing@sublimesecurity.com\",\"local_part\":\"testing\"},\"ips\":[{\"ip\":\"1.128.0.0\"}],\"mailer\":\"MyCustomMailer\",\"message_id\":\"2fe271830bbad5fe3a70abbe7a8c0bfe7refe3ffe\",\"domains\":[{\"domain\":\"test.com\",\"subdomain\":\"test\",\"tld\":\"com\",\"punycode\":\"test.com\",\"root_domain\":\"test.com\",\"sld\":\"test\",\"valid\":true},{\"domain\":\"example.com\",\"subdomain\":\"example\",\"tld\":\"com\",\"punycode\":\"example.com\",\"root_domain\":\"example.com\",\"sld\":\"example\",\"valid\":true}],\"reply_to\":[{\"email\":{\"email\":\"user@example.com\",\"local_part\":\"user\",\"domain\":{\"domain\":\"example.com\",\"root_domain\":\"example.com\",\"sld\":\"example\",\"tld\":\"com\",\"valid\":true}}},{\"display_name\":\"Example Display Name\",\"email\":{\"domain\":{\"punycode\":\"example.com\",\"subdomain\":\"sub.example\"}}},{\"display_name\":\"Another Display Name\",\"email\":{\"domain\":{\"punycode\":\"anotherexample.com\",\"subdomain\":\"sub.anotherexample\"}}}]},\"type\":{\"outbound\":true},\"mailbox\":{\"email\":{\"email\":\"user@example.com\",\"local_part\":\"user\",\"domain\":{\"domain\":\"example.com\",\"root_domain\":\"example.com\",\"sld\":\"example\",\"tld\":\"com\",\"valid\":true,\"punycode\":\"xn--example-d4a.com\",\"subdomain\":\"sub\"}}},\"recipients\":{\"to\":[{\"display_name\":\"Alice Johnson\",\"email\":{\"email\":\"user@example.com\",\"local_part\":\"user\",\"domain\":{\"domain\":\"example.com\",\"root_domain\":\"example.com\",\"sld\":\"example\",\"tld\":\"com\",\"valid\":true,\"punycode\":\"xn--example-d4a.net\",\"subdomain\":\"sub\"}}}],\"bcc\":[{\"display_name\":\"John Doe\",\"email\":{\"domain\":{\"domain\":\"example.com\",\"punycode\":\"xn--example-d4a.com\",\"root_domain\":\"example\",\"sld\":\"example\",\"subdomain\":\"sub\",\"tld\":\"com\",\"valid\":true},\"email\":\"john.doe@example.com\",\"local_part\":\"john.doe\"}}],\"cc\":[{\"display_name\":\"Jane Smith\",\"email\":{\"domain\":{\"domain\":\"example.org\",\"punycode\":\"xn--example-d4a.org\",\"root_domain\":\"example\",\"sld\":\"example\",\"subdomain\":\"sub\",\"tld\":\"org\",\"valid\":true},\"email\":\"jane.smith@example.org\",\"local_part\":\"jane.smith\"}}]},\"sender\":{\"display_name\":\"Sublime Security Test\",\"email\":{\"email\":\"testing@sublimesecurity.com\",\"local_part\":\"testing\",\"domain\":{\"domain\":\"sublimesecurity.com\",\"root_domain\":\"sublimesecurity.com\",\"sld\":\"sublimesecurity\",\"tld\":\"com\",\"valid\":true,\"punycode\":\"xn--example-d4a.com\",\"subdomain\":\"sub\"}}},\"subject\":{\"subject\":\"Sublime-Security-Standard-Test-String\"},\"_meta\":{\"id\":\"01911208-633c-7f03-b303-e594d92cf818\",\"canonical_id\":\"2fe271830bbad5fe3a70abbe7a8c0bfe79eb208a76cde267930d19f0e8cea81c\",\"created_at\":\"2024-08-02T07:40:25.135939305Z\",\"effective_at\":\"2024-08-02T07:40:25.135939305Z\"},\"_errors\":[{\"field\":\"Mime-Version\",\"message\":\"No Mime-Version defined in headers\",\"type\":\"missing_header_field\"}]}", + "type": [ + "info" + ] + }, + "input": { + "type": "aws-s3" + }, + "log": { + "file": { + "path": "https://elastic-package-sublime-security-bucket-33881.s3.us-east-1.amazonaws.com/email-message.log" + }, + "offset": 0 + }, + "observer": { + "product": "Sublime Security", + "vendor": "Sublime Security" + }, + "related": { + "hash": [ + "1a2b3c", + "7h8i9j", + "1x2y3z", + "4d5e6f", + "1k2l3m", + "4a5b6c", + "7g8h9i", + "4n5o6p", + "7d8e9f", + "abcdefg" + ], + "hosts": [ + "example.com", + "mail.example.com", + "test.com", + "example", + "example.org", + "sublimesecurity.com" + ], + "ip": [ + "1.128.0.0" + ], + "user": [ + "test", + "user@example.com", + "john.doe@example.com", + "jane.smith@example.org", + "testing@sublimesecurity.com" + ] + }, + "source": { + "domain": "sublimesecurity.com", + "ip": "1.128.0.0", + "subdomain": "sub", + "top_level_domain": "com" + }, + "sublime_security": { + "email_message": { + "attachments": [ + { + "content": { + "id": "abc123", + "transfer_encoding": "base64" + }, + "file": { + "type": "document" + }, + "raw": "JVBERi0xLjMKJcfs4AAQSkZjRgABAQE" + }, + { + "content": { + "id": "xyz456", + "transfer_encoding": "7bit" + }, + "file": { + "type": "image" + }, + "raw": "/9j/4AAQSkZJRgABAQEJVBERi0xLjMKJd" + }, + { + "content": { + "id": "efg789", + "transfer_encoding": "quoted-printable" + }, + "file": { + "type": "text" + }, + "raw": "SGVsbG8gdVsbG8gd29yb29ybGQhVsbG8gd29yb" + } + ], + "body": { + "current_thread": { + "text": "Sublime Security test message." + }, + "html": { + "charset": "utf-8", + "content_transfer_encoding": "base64", + "display_text": "Sublime Security test message.", + "inner_text": "

Sublime Security test message.

", + "raw": "

Sublime Security test message.

" + }, + "ips": [ + { + "ip": "1.128.0.0" + } + ], + "links": [ + { + "display_text": "Click here!", + "display_url": { + "domain": { + "domain": "example.com", + "punycode": "demo", + "root_domain": "example.com", + "sld": "example", + "subdomain": "example", + "tld": "com", + "valid": true + }, + "fragment": "search", + "password": "pass123", + "path": "/test", + "port": 80, + "query_params": "q=elasticsearch", + "rewrite": { + "encoders": [ + "base64" + ], + "original": "demo" + }, + "scheme": "https", + "url": "https://example.com/test?q=elasticsearch#search", + "username": "test" + }, + "mismatched": true + }, + { + "href_url": { + "domain": { + "punycode": "demo", + "root_domain": "example.com", + "sld": "example", + "valid": true + }, + "rewrite": { + "encoders": [ + "base64" + ], + "original": "demo" + } + } + } + ], + "plain": { + "charset": "utf-8", + "content_transfer_encoding": "base64", + "raw": "Sublime Security test message.\n" + } + }, + "errors": [ + { + "field": "Mime-Version", + "message": "No Mime-Version defined in headers", + "type": "missing_header_field" + } + ], + "external": { + "message_id": "2fe271830bbad5fe3a70abbe7a8c0bfe7refe3ffe", + "route_type": "sent", + "spam": false, + "spam_folder": true, + "thread_id": "sample_data" + }, + "headers": { + "auth_summary": { + "dmarc": { + "details": { + "action": "quarantine", + "disposition": "quarantine", + "from": { + "domain": "example.com", + "punycode": "example.com", + "root_domain": "example.com", + "sld": "example", + "subdomain": "example", + "tld": "com", + "valid": true + }, + "policy": "reject", + "sub_policy": "none", + "verdict": "pass", + "version": "1.0" + }, + "pass": true, + "received_hop": 1 + }, + "spf": { + "details": { + "client_ip": { + "ip": "1.128.0.0" + }, + "description": "SPF record found", + "designator": "pass", + "helo": { + "domain": "example.com", + "punycode": "example.com", + "root_domain": "example.com", + "sld": "example", + "subdomain": "example", + "tld": "com", + "valid": true + }, + "server": { + "domain": "mail.example.com", + "punycode": "mail.example.com", + "root_domain": "example.com", + "sld": "example", + "subdomain": "mail", + "tld": "com", + "valid": true + }, + "verdict": "pass" + }, + "error": true, + "pass": true, + "received_hop": 2 + } + }, + "date": "2019-10-21T18:23:24.000Z", + "date_original_offset": "-4", + "delivered_to": { + "domain": { + "punycode": "example.com", + "root_domain": "example.com", + "sld": "example", + "valid": true + }, + "email": "testing@sublimesecurity.com", + "local_part": "testing" + }, + "domains": [ + { + "domain": "test.com", + "punycode": "test.com", + "root_domain": "test.com", + "sld": "test", + "subdomain": "test", + "tld": "com", + "valid": true + }, + { + "domain": "example.com", + "punycode": "example.com", + "root_domain": "example.com", + "sld": "example", + "subdomain": "example", + "tld": "com", + "valid": true + } + ], + "hops": [ + { + "authentication_results": { + "compauth": { + "reason": "reason_value", + "verdict": "pass" + }, + "dkim": "pass", + "dkim_details": { + "algorithm": "rsa-sha256", + "body_hash": "abcdefg", + "domain": "example.com", + "headers": "from, to, subject", + "instance": "example.com", + "selector": "abcdefg", + "signature": "abcdefg", + "type": "dkim", + "version": "1.0" + }, + "dmarc": "pass", + "dmarc_details": { + "action": "quarantine", + "disposition": "quarantine", + "from": { + "domain": "example.com", + "punycode": "example.com", + "root_domain": "example.com", + "sld": "example", + "subdomain": "example", + "tld": "com", + "valid": true + }, + "policy": "reject", + "sub_policy": "none", + "verdict": "pass", + "version": "1.0" + }, + "instance": "example.com", + "server": { + "domain": "mail.example.com", + "punycode": "mail.example.com", + "root_domain": "example.com", + "sld": "example", + "subdomain": "mail", + "tld": "com", + "valid": true + }, + "spf": "pass", + "spf_details": { + "client_ip": { + "ip": "1.128.0.0" + }, + "description": "SPF record found", + "designator": "pass", + "helo": { + "domain": "example.com", + "punycode": "example.com", + "root_domain": "example.com", + "sld": "example", + "subdomain": "example", + "tld": "com", + "valid": true + }, + "server": { + "domain": "mail.example.com", + "root_domain": "example.com", + "sld": "example", + "subdomain": "mail", + "tld": "com", + "valid": true + }, + "verdict": "pass" + }, + "type": "spf" + }, + "fields": [ + { + "name": "To", + "position": 0, + "to": "user@example.com", + "value": "user@example.com" + }, + { + "name": "Subject", + "position": 1, + "subject": "Sublime-Security-Standard-Test-String", + "value": "Sublime-Security-Standard-Test-String" + }, + { + "date": "Mon, 21 Oct 2019 14:23:24 -0400", + "name": "Date", + "position": 2, + "value": "Mon, 21 Oct 2019 14:23:24 -0400" + }, + { + "from": "Sublime Security Test ", + "name": "From", + "position": 3, + "value": "Sublime Security Test " + } + ], + "index": 0, + "received": { + "additional": { + "raw": "Authentication successful" + }, + "id": { + "raw": "msg-12345" + }, + "link": { + "raw": "https://mail.example.com/message/12345" + }, + "mailbox": { + "raw": "user@example.com" + }, + "protocol": { + "raw": "IMAP" + }, + "server": { + "raw": "imap.example.com" + }, + "source": { + "raw": "81.2.69.144" + }, + "time": "2019-10-21T18:23:24.000Z", + "zone_offset": "+00:00" + }, + "received_spf": { + "client_ip": { + "ip": "1.128.0.0" + }, + "description": "SPF record found", + "designator": "pass", + "helo": { + "domain": "example.com", + "punycode": "example.com", + "root_domain": "example.com", + "sld": "example", + "subdomain": "example", + "tld": "com", + "valid": true + }, + "server": { + "domain": "mail.example.com", + "punycode": "mail.example.com", + "root_domain": "example.com", + "sld": "example", + "subdomain": "mail", + "tld": "com", + "valid": true + }, + "verdict": "pass" + }, + "signature": { + "algorithm": "rsa-sha256", + "body_hash": "b9c4a3f9d93d9a38bdf8c47a8f2d2c79ec1d8b1f", + "domain": "example.com", + "headers": "from:to:subject:date", + "instance": "123456", + "selector": "default", + "signature": "d2abf9d6c8f4b8d68d8f3f7b6f9d3b8e6a8c2b3a9f4b8d7b9d3b6a8f9c3b4e5f", + "type": "spf", + "version": "1" + } + } + ], + "in_reply_to": "in_reply_to_value", + "ips": [ + { + "ip": "1.128.0.0" + } + ], + "references": [ + "test1", + "test2" + ], + "reply_to": [ + { + "email": { + "domain": { + "domain": "example.com", + "root_domain": "example.com", + "sld": "example", + "tld": "com", + "valid": true + }, + "local_part": "user" + } + }, + { + "display_name": "Example Display Name", + "email": { + "domain": { + "punycode": "example.com", + "subdomain": "sub.example" + } + } + }, + { + "display_name": "Another Display Name", + "email": { + "domain": { + "punycode": "anotherexample.com", + "subdomain": "sub.anotherexample" + } + } + } + ], + "return_path": { + "domain": { + "domain": "example.com", + "punycode": "xn--example-d4a.com", + "root_domain": "example", + "sld": "example", + "subdomain": "sub", + "tld": "com", + "valid": true + }, + "email": "user@example.com", + "local_part": "user" + }, + "x_authenticated_domain": { + "domain": "example.com", + "punycode": "xn--example-d4a.com", + "root_domain": "example.com", + "sld": "example", + "subdomain": "sub", + "tld": "com", + "valid": true + }, + "x_authenticated_sender": { + "domain": { + "domain": "example.com", + "punycode": "example.com", + "root_domain": "example.com", + "sld": "example", + "subdomain": "sub", + "tld": "com", + "valid": true + }, + "email": "user@example.com", + "local_part": "user" + }, + "x_originating_ip": { + "ip": "1.128.0.0" + }, + "x_secure_server_account": "account_value", + "x_sender": { + "domain": { + "domain": "example.com", + "punycode": "example.com", + "root_domain": "example.com", + "sld": "example", + "subdomain": "sub", + "tld": "com", + "valid": true + }, + "email": "user@example.com", + "local_part": "user" + } + }, + "mailbox": { + "email": { + "domain": { + "domain": "example.com", + "punycode": "xn--example-d4a.com", + "root_domain": "example.com", + "sld": "example", + "subdomain": "sub", + "tld": "com", + "valid": true + }, + "local_part": "user", + "value": "user@example.com" + } + }, + "meta": { + "canonical_id": "2fe271830bbad5fe3a70abbe7a8c0bfe79eb208a76cde267930d19f0e8cea81c", + "effective_at": "2024-08-02T07:40:25.135Z" + }, + "recipients": { + "bcc": [ + { + "display_name": "John Doe", + "email": { + "domain": { + "domain": "example.com", + "punycode": "xn--example-d4a.com", + "root_domain": "example", + "sld": "example", + "subdomain": "sub", + "tld": "com", + "valid": true + }, + "local_part": "john.doe" + } + } + ], + "cc": [ + { + "display_name": "Jane Smith", + "email": { + "domain": { + "domain": "example.org", + "punycode": "xn--example-d4a.org", + "root_domain": "example", + "sld": "example", + "subdomain": "sub", + "tld": "org", + "valid": true + }, + "local_part": "jane.smith" + } + } + ], + "to": [ + { + "display_name": "Alice Johnson", + "email": { + "domain": { + "domain": "example.com", + "punycode": "xn--example-d4a.net", + "root_domain": "example.com", + "sld": "example", + "subdomain": "sub", + "tld": "com", + "valid": true + }, + "local_part": "user" + } + } + ] + }, + "sender": { + "display_name": "Sublime Security Test", + "email": { + "domain": { + "punycode": "xn--example-d4a.com", + "root_domain": "sublimesecurity.com", + "sld": "sublimesecurity", + "valid": true + }, + "local_part": "testing" + } + }, + "type": { + "outbound": true + } + } + }, + "tags": [ + "collect_sqs_logs", + "preserve_original_event", + "forwarded", + "sublime_security-email_message" + ], + "url": [ + { + "domain": "example.com", + "fragment": "search", + "full": "https://example.com/test?q=elasticsearch#search", + "password": "pass123", + "path": "/test", + "port": 80, + "query": "q=elasticsearch", + "scheme": "https", + "subdomain": "example", + "top_level_domain": "com", + "username": "test" + } + ], + "user_agent": { + "device": { + "name": "Other" + }, + "name": "Other", + "original": "MyCustomMailer" + } +} \ No newline at end of file diff --git a/packages/sublime_security/data_stream/message_event/_dev/deploy/docker/docker-compose.yml b/packages/sublime_security/data_stream/message_event/_dev/deploy/docker/docker-compose.yml new file mode 100644 index 00000000000..979573f79b3 --- /dev/null +++ b/packages/sublime_security/data_stream/message_event/_dev/deploy/docker/docker-compose.yml @@ -0,0 +1,15 @@ +version: '2.3' +services: + sublime_security: + image: docker.elastic.co/observability/stream:v0.15.0 + hostname: sublime_security + ports: + - 8090 + volumes: + - ./files:/files:ro + environment: + PORT: '8090' + command: + - http-server + - --addr=:8090 + - --config=/files/config.yml diff --git a/packages/sublime_security/data_stream/message_event/_dev/deploy/docker/files/config.yml b/packages/sublime_security/data_stream/message_event/_dev/deploy/docker/files/config.yml new file mode 100644 index 00000000000..190051bb6d2 --- /dev/null +++ b/packages/sublime_security/data_stream/message_event/_dev/deploy/docker/files/config.yml @@ -0,0 +1,160 @@ +rules: + - path: /v0/message-groups + methods: ['GET'] + query_params: + created_at__gte: "{created_at__gte:.*}" + created_at__lt: "{created_at__lt:.*}" + flagged: "{flagged:.*}" + limit: 1 + offset: 0 + request_headers: + Authorization: + - "Bearer xxxx" + responses: + - status_code: 200 + headers: + Content-Type: + - 'application/json' + body: |- + {{ minify_json ` + { + "message_groups": [ + { + "messages": [ + { + "id": "9c426680-5cdf-4283-adbd-d79ba0e52434" + }, + { + "id": "26bf7e5c-4b46-4042-90de-eacb22dc1b3d" + } + ] + } + ], + "count": 1, + "total": 1, + "stats_limit_exceeded": false + } + `}} + - path: /v0/message-groups + methods: ['GET'] + query_params: + created_at__gte: "{created_at__gte:.*}" + created_at__lt: "{created_at__lt:.*}" + flagged: "{flagged:.*}" + limit: 1 + offset: 1 + request_headers: + Authorization: + - "Bearer xxxx" + responses: + - status_code: 200 + headers: + Content-Type: + - 'application/json' + body: |- + {{ minify_json ` + { + "message_groups": [], + "count": 0, + "total": 1, + "stats_limit_exceeded": false + } + `}} + - path: /v0/messages/9c426680-5cdf-4283-adbd-d79ba0e52434 + methods: ['GET'] + request_headers: + Authorization: + - "Bearer xxxx" + responses: + - status_code: 200 + headers: + Content-Type: + - 'application/json' + body: |- + {{ minify_json ` + { + "id": "9c426680-5cdf-4283-adbd-d79ba0e52434", + "canonical_id": "dd97dc82731ff7e82edfccaef59826cccd271bd4423e09d1e150ade83037cb37", + "created_at": "2024-07-12T05:15:08.221838Z", + "external_id": "7a2dfbeb-1310-48fc-9ed9-f480608a0306", + "message_source_id": "257982a1-f106-4c68-bc64-ff032914ed5f", + "mailbox": { + "id": "433fe142-e2e5-4372-84ea-480279543a9b", + "external_id": null, + "email": "demo@example.com" + }, + "subject": "Urgent: Wire transfer", + "sender": { + "email": "bob.demo@gmail.com", + "display_name": "Bob Doe" + }, + "landed_in_spam": false, + "read_at": null, + "replied_at": null, + "forwarded_at": null, + "forward_recipients": [], + "recipients": [ + { + "email": "xyz@example.com" + }, + { + "email": "user12@example.com" + }, + { + "email": "user@example.com" + }, + { + "email": "leon12@example.com" + } + ] + } + `}} + - path: /v0/messages/26bf7e5c-4b46-4042-90de-eacb22dc1b3d + methods: ['GET'] + request_headers: + Authorization: + - "Bearer xxxx" + responses: + - status_code: 200 + headers: + Content-Type: + - 'application/json' + body: |- + {{ minify_json ` + { + "id": "26bf7e5c-4b46-4042-90de-eacb22dc1b3d", + "canonical_id": "dd97dc82731ff7e82edfccaef59826cccd271bd4423e09d1e150ade83037cb37", + "created_at": "2024-07-12T05:15:10.447703Z", + "external_id": "88557fd9-0dee-4171-998e-9a949f01e1f5", + "message_source_id": "257982a1-f106-4c68-bc64-ff032914ed5f", + "mailbox": { + "id": "aae02fa3-5693-4ca5-80d0-15e082eff3bc", + "external_id": null, + "email": "user@example.com" + }, + "subject": "Urgent: Wire transfer", + "sender": { + "email": "alice.demo@example.com", + "display_name": "Alice Doe" + }, + "landed_in_spam": false, + "read_at": null, + "replied_at": null, + "forwarded_at": null, + "forward_recipients": [], + "recipients": [ + { + "email": "abc@example.com" + }, + { + "email": "xyz@example.com" + }, + { + "email": "john@example.com" + }, + { + "email": "leon@example.com" + } + ] + } + `}} \ No newline at end of file diff --git a/packages/sublime_security/data_stream/message_event/_dev/test/pipeline/test-common-config.yml b/packages/sublime_security/data_stream/message_event/_dev/test/pipeline/test-common-config.yml new file mode 100644 index 00000000000..37e8fa225fd --- /dev/null +++ b/packages/sublime_security/data_stream/message_event/_dev/test/pipeline/test-common-config.yml @@ -0,0 +1,3 @@ +fields: + tags: + - preserve_duplicate_custom_fields diff --git a/packages/sublime_security/data_stream/message_event/_dev/test/pipeline/test-message-event.log b/packages/sublime_security/data_stream/message_event/_dev/test/pipeline/test-message-event.log new file mode 100644 index 00000000000..52c0081ecc9 --- /dev/null +++ b/packages/sublime_security/data_stream/message_event/_dev/test/pipeline/test-message-event.log @@ -0,0 +1,2 @@ +{"id":"ef875384-6868-4e60-819a-e392f3fc1344","canonical_id":"ae608db67f666a7c92729c4a018e24362caea4c3b4b9fa00534a838dc989bab3","created_at":"2024-07-12T05:15:19.061701Z","external_id":"811a4cb9-aa1c-4e81-a0d1-21e163872c08","message_source_id":"257982a1-f106-4c68-bc64-ff032914ed5f","mailbox":{"id":"433fe142-e2e5-4372-84ea-480279543a9b","external_id":null,"email":"alice@example.com"},"subject":"Available?","sender":{"email":"alice@example.com","display_name":"Alice"},"landed_in_spam":false,"read_at":null,"replied_at":null,"forwarded_at":null,"forward_recipients":[],"recipients":[{"email":"bob@example.com"},{"email":"john@example.com"}]} +{"created_at":"2023-05-04T01:06:50.73801Z","data":{"flagged_rules":[{"id":"958ad820-767f-4c83-ac44-94032151574f","name":"Test rule1","severity":"high","tags":["Testing"]},{"id":"2483da42-6905-4eca-bc8e-a5475ab906a4","name":"Test rule2","severity":"high","tags":["Testing"]},{"id":"9f95f7bb-b1f1-465a-a7a4-23266792c000","name":"Test rule3","severity":"low","tags":["Testing"]},{"id":"862b4f5b-c0f7-4e36-89b2-3091f9ac492b","name":"Test rule4","severity":"medium","tags":["Testing"]}],"message":{"canonical_id":"8fdb3acd8441b18478b7e80927c732f44e9086dcf7cfa6c1186ed21637427ef8","external_id":"187e44c2288f2e8d","id":"089b2e97-2107-4d1e-83e1-1920314dbb80","mailbox":{"external_id":"100712120249161713725","id":"3142d2f8-ac0d-4e37-bb1f-d5e4ff5f7f96"},"message_source_id":"d76ce576-4687-4182-b8e3-a220d6878b21"},"triggered_actions":[{"id":"9f95f7bb-b1f1-465a-a7a4-23266792c000","name":"test","type":"alert"},{"id":"9f95f7bb-b1f1-465a-a7a4-23266792c001","name":"test 1","type":"rule"}]},"type":"message.flagged"} diff --git a/packages/sublime_security/data_stream/message_event/_dev/test/pipeline/test-message-event.log-expected.json b/packages/sublime_security/data_stream/message_event/_dev/test/pipeline/test-message-event.log-expected.json new file mode 100644 index 00000000000..8aa37d5d26b --- /dev/null +++ b/packages/sublime_security/data_stream/message_event/_dev/test/pipeline/test-message-event.log-expected.json @@ -0,0 +1,182 @@ +{ + "expected": [ + { + "@timestamp": "2024-07-12T05:15:19.061Z", + "ecs": { + "version": "8.11.0" + }, + "email": { + "from": { + "address": [ + "alice@example.com" + ] + }, + "subject": "Available?", + "to": { + "address": [ + "bob@example.com", + "john@example.com" + ] + } + }, + "event": { + "category": [ + "email" + ], + "id": "ef875384-6868-4e60-819a-e392f3fc1344", + "kind": "event", + "original": "{\"id\":\"ef875384-6868-4e60-819a-e392f3fc1344\",\"canonical_id\":\"ae608db67f666a7c92729c4a018e24362caea4c3b4b9fa00534a838dc989bab3\",\"created_at\":\"2024-07-12T05:15:19.061701Z\",\"external_id\":\"811a4cb9-aa1c-4e81-a0d1-21e163872c08\",\"message_source_id\":\"257982a1-f106-4c68-bc64-ff032914ed5f\",\"mailbox\":{\"id\":\"433fe142-e2e5-4372-84ea-480279543a9b\",\"external_id\":null,\"email\":\"alice@example.com\"},\"subject\":\"Available?\",\"sender\":{\"email\":\"alice@example.com\",\"display_name\":\"Alice\"},\"landed_in_spam\":false,\"read_at\":null,\"replied_at\":null,\"forwarded_at\":null,\"forward_recipients\":[],\"recipients\":[{\"email\":\"bob@example.com\"},{\"email\":\"john@example.com\"}]}", + "type": [ + "info" + ] + }, + "observer": { + "product": "Sublime Security", + "vendor": "Sublime Security" + }, + "related": { + "user": [ + "bob@example.com", + "john@example.com", + "Alice", + "alice@example.com" + ] + }, + "source": { + "user": { + "name": "Alice" + } + }, + "sublime_security": { + "message_event": { + "canonical_id": "ae608db67f666a7c92729c4a018e24362caea4c3b4b9fa00534a838dc989bab3", + "created_at": "2024-07-12T05:15:19.061Z", + "external_id": "811a4cb9-aa1c-4e81-a0d1-21e163872c08", + "id": "ef875384-6868-4e60-819a-e392f3fc1344", + "landed_in_spam": false, + "mailbox": { + "email": "alice@example.com", + "id": "433fe142-e2e5-4372-84ea-480279543a9b" + }, + "message_source_id": "257982a1-f106-4c68-bc64-ff032914ed5f", + "recipients": [ + { + "email": "bob@example.com" + }, + { + "email": "john@example.com" + } + ], + "sender": { + "display_name": "Alice", + "email": "alice@example.com" + }, + "subject": "Available?" + } + }, + "tags": [ + "preserve_duplicate_custom_fields" + ] + }, + { + "@timestamp": "2023-05-04T01:06:50.738Z", + "ecs": { + "version": "8.11.0" + }, + "event": { + "category": [ + "email" + ], + "id": "089b2e97-2107-4d1e-83e1-1920314dbb80", + "kind": "event", + "original": "{\"created_at\":\"2023-05-04T01:06:50.73801Z\",\"data\":{\"flagged_rules\":[{\"id\":\"958ad820-767f-4c83-ac44-94032151574f\",\"name\":\"Test rule1\",\"severity\":\"high\",\"tags\":[\"Testing\"]},{\"id\":\"2483da42-6905-4eca-bc8e-a5475ab906a4\",\"name\":\"Test rule2\",\"severity\":\"high\",\"tags\":[\"Testing\"]},{\"id\":\"9f95f7bb-b1f1-465a-a7a4-23266792c000\",\"name\":\"Test rule3\",\"severity\":\"low\",\"tags\":[\"Testing\"]},{\"id\":\"862b4f5b-c0f7-4e36-89b2-3091f9ac492b\",\"name\":\"Test rule4\",\"severity\":\"medium\",\"tags\":[\"Testing\"]}],\"message\":{\"canonical_id\":\"8fdb3acd8441b18478b7e80927c732f44e9086dcf7cfa6c1186ed21637427ef8\",\"external_id\":\"187e44c2288f2e8d\",\"id\":\"089b2e97-2107-4d1e-83e1-1920314dbb80\",\"mailbox\":{\"external_id\":\"100712120249161713725\",\"id\":\"3142d2f8-ac0d-4e37-bb1f-d5e4ff5f7f96\"},\"message_source_id\":\"d76ce576-4687-4182-b8e3-a220d6878b21\"},\"triggered_actions\":[{\"id\":\"9f95f7bb-b1f1-465a-a7a4-23266792c000\",\"name\":\"test\",\"type\":\"alert\"},{\"id\":\"9f95f7bb-b1f1-465a-a7a4-23266792c001\",\"name\":\"test 1\",\"type\":\"rule\"}]},\"type\":\"message.flagged\"}", + "type": [ + "info" + ] + }, + "observer": { + "product": "Sublime Security", + "vendor": "Sublime Security" + }, + "rule": { + "id": [ + "958ad820-767f-4c83-ac44-94032151574f", + "2483da42-6905-4eca-bc8e-a5475ab906a4", + "9f95f7bb-b1f1-465a-a7a4-23266792c000", + "862b4f5b-c0f7-4e36-89b2-3091f9ac492b" + ], + "name": [ + "Test rule1", + "Test rule2", + "Test rule3", + "Test rule4" + ] + }, + "sublime_security": { + "message_event": { + "canonical_id": "8fdb3acd8441b18478b7e80927c732f44e9086dcf7cfa6c1186ed21637427ef8", + "created_at": "2023-05-04T01:06:50.738Z", + "data": { + "flagged_rules": [ + { + "id": "958ad820-767f-4c83-ac44-94032151574f", + "name": "Test rule1", + "severity": "high", + "tags": [ + "Testing" + ] + }, + { + "id": "2483da42-6905-4eca-bc8e-a5475ab906a4", + "name": "Test rule2", + "severity": "high", + "tags": [ + "Testing" + ] + }, + { + "id": "9f95f7bb-b1f1-465a-a7a4-23266792c000", + "name": "Test rule3", + "severity": "low", + "tags": [ + "Testing" + ] + }, + { + "id": "862b4f5b-c0f7-4e36-89b2-3091f9ac492b", + "name": "Test rule4", + "severity": "medium", + "tags": [ + "Testing" + ] + } + ], + "triggered_actions": [ + { + "id": "9f95f7bb-b1f1-465a-a7a4-23266792c000", + "name": "test", + "type": "alert" + }, + { + "id": "9f95f7bb-b1f1-465a-a7a4-23266792c001", + "name": "test 1", + "type": "rule" + } + ] + }, + "external_id": "187e44c2288f2e8d", + "id": "089b2e97-2107-4d1e-83e1-1920314dbb80", + "mailbox": { + "external_id": "100712120249161713725", + "id": "3142d2f8-ac0d-4e37-bb1f-d5e4ff5f7f96" + }, + "message_source_id": "d76ce576-4687-4182-b8e3-a220d6878b21", + "type": "message.flagged" + } + }, + "tags": [ + "preserve_duplicate_custom_fields" + ] + } + ] +} \ No newline at end of file diff --git a/packages/sublime_security/data_stream/message_event/_dev/test/system/test-cel-config.yml b/packages/sublime_security/data_stream/message_event/_dev/test/system/test-cel-config.yml new file mode 100644 index 00000000000..6d2105a4317 --- /dev/null +++ b/packages/sublime_security/data_stream/message_event/_dev/test/system/test-cel-config.yml @@ -0,0 +1,14 @@ +input: cel +service: sublime_security +vars: + url: http://{{Hostname}}:{{Port}} + api_key: xxxx +data_stream: + vars: + interval: 5m + initial_interval: 24h + page_size: 1 + preserve_original_event: true + preserve_duplicate_custom_fields: true +assert: + hit_count: 2 diff --git a/packages/sublime_security/data_stream/message_event/agent/stream/aws-s3.yml.hbs b/packages/sublime_security/data_stream/message_event/agent/stream/aws-s3.yml.hbs new file mode 100644 index 00000000000..7bf0e86682e --- /dev/null +++ b/packages/sublime_security/data_stream/message_event/agent/stream/aws-s3.yml.hbs @@ -0,0 +1,97 @@ +{{#if collect_s3_logs}} + +{{#if bucket_arn}} +bucket_arn: {{bucket_arn}} +{{/if}} +{{#if number_of_workers}} +number_of_workers: {{number_of_workers}} +{{/if}} +{{#if interval}} +bucket_list_interval: {{interval}} +{{/if}} +{{#if bucket_list_prefix}} +bucket_list_prefix: {{bucket_list_prefix}} +{{/if}} + +{{else}} + +{{#if queue_url}} +queue_url: {{queue_url}} +{{/if}} +{{#if region}} +region: {{region}} +{{/if}} +{{#if visibility_timeout}} +visibility_timeout: {{visibility_timeout}} +{{/if}} +{{#if api_timeout}} +api_timeout: {{api_timeout}} +{{/if}} +{{#if max_number_of_messages}} +max_number_of_messages: {{max_number_of_messages}} +{{/if}} +{{#if file_selectors}} +file_selectors: +{{file_selectors}} +{{/if}} + +{{/if}} + +expand_event_list_from_field: events +content_type: application/json +{{#if access_key_id}} +access_key_id: {{access_key_id}} +{{/if}} +{{#if secret_access_key}} +secret_access_key: {{secret_access_key}} +{{/if}} +{{#if session_token}} +session_token: {{session_token}} +{{/if}} +{{#if shared_credential_file}} +shared_credential_file: {{shared_credential_file}} +{{/if}} +{{#if credential_profile_name}} +credential_profile_name: {{credential_profile_name}} +{{/if}} +{{#if role_arn}} +role_arn: {{role_arn}} +{{/if}} +{{#if external_id}} +external_id: {{external_id}} +{{/if}} +{{#if default_region}} +default_region: {{default_region}} +{{/if}} + +{{#if fips_enabled}} +fips_enabled: {{fips_enabled}} +{{/if}} +{{#if proxy_url}} +proxy_url: {{proxy_url}} +{{/if}} +{{#if ssl}} +ssl: {{ssl}} +{{/if}} +tags: +{{#if collect_s3_logs}} + - collect_s3_logs +{{else}} + - collect_sqs_logs +{{/if}} +{{#if preserve_original_event}} + - preserve_original_event +{{/if}} +{{#if preserve_duplicate_custom_fields}} + - preserve_duplicate_custom_fields +{{/if}} +{{#each tags as |tag|}} + - {{tag}} +{{/each}} +{{#contains "forwarded" tags}} +publisher_pipeline.disable_host: true +{{/contains}} +{{#if processors}} +processors: +{{processors}} +{{/if}} diff --git a/packages/sublime_security/data_stream/message_event/agent/stream/cel.yml.hbs b/packages/sublime_security/data_stream/message_event/agent/stream/cel.yml.hbs new file mode 100644 index 00000000000..38a2113a64d --- /dev/null +++ b/packages/sublime_security/data_stream/message_event/agent/stream/cel.yml.hbs @@ -0,0 +1,148 @@ +config_version: 2 +interval: {{interval}} +{{#if enable_request_tracer}} +resource.tracer.filename: "../../logs/cel/http-request-trace-*.ndjson" +resource.tracer.maxbackups: 5 +{{/if}} +{{#if proxy_url}} +resource.proxy_url: {{proxy_url}} +{{/if}} +{{#if ssl}} +resource.ssl: {{ssl}} +{{/if}} +{{#if http_client_timeout}} +resource.timeout: {{http_client_timeout}} +{{/if}} +resource.url: {{url}} +state: + page_size: {{page_size}} + offset: 0 + initial_interval: {{initial_interval}} + api_key: {{api_key}} +redact: + fields: + - api_key +program: | + ( + has(state.worklist) && size(state.worklist) > 0 ? + state + : + ( + state.?want_more.orValue(false) ? + state + : + state.with({ + "start_time": state.?cursor.last_timestamp.orValue((now - duration(state.initial_interval)).format(time_layout.RFC3339)), + "end_time": now.format(time_layout.RFC3339), + }) + ).as(state, state.with( + request( + "GET", + state.url.trim_right("/") + "/v0/message-groups?" + { + "created_at__gte": [state.start_time], + "created_at__lt": [state.end_time], + "flagged": ["true"], + "limit": [string(state.page_size)], + "offset": [string(state.offset)] + }.format_query() + ).with({ + "Header": { + "Authorization": ["Bearer " + state.api_key], + }, + }).do_request().as(resp, resp.StatusCode == 200 ? + ( + bytes(resp.Body).decode_json().as(body,{ + "worklist": size(body.message_groups) > 0 ? + body.message_groups.map(group, + has(group.messages) && size(group.messages) > 0 ? group.messages.map(e, e.id) : [] + ).flatten() + : + [], + "next": 0, + }) + ) + : + { + "events": { + "error": { + "code": string(resp.StatusCode), + "id": string(resp.Status), + "message": "GET:"+( + size(resp.Body) != 0 ? + string(resp.Body) + : + string(resp.Status) + ' (' + string(resp.StatusCode) + ')' + ), + }, + }, + "want_more": false, + } + ) + )) + ).as(state, state.with( + !has(state.worklist) ? state : + int(state.next) < size(state.worklist) ? + request( + "GET", + state.url.trim_right("/") + "/v0/messages/" + state.worklist[state.next] + ).with({ + "Header": { + "Authorization": ["Bearer " + string(state.api_key)], + }, + }).do_request().as(resp,resp.StatusCode == 200 ? + bytes(resp.Body).decode_json().as(body,{ + "events": [{ + "message": body.encode_json(), + }], + "cursor": { + "last_timestamp": state.end_time + }, + "worklist": (int(state.next) + 1) < size(state.worklist) ? state.worklist : [], + "next": (int(state.next) + 1) < size(state.worklist) ? (int(state.next) + 1 ) : 0, + "offset": (int(state.next) + 1) < size(state.worklist) ? state.offset : int(state.offset) + int(state.page_size), + "want_more": true, + }) + : + { + "events": { + "error": { + "code": string(resp.StatusCode), + "id": string(resp.Status), + "message": "GET:"+( + size(resp.Body) != 0 ? + string(resp.Body) + : + string(resp.Status) + ' (' + string(resp.StatusCode) + ')' + ), + }, + }, + "want_more": false, + } + ) + : + { + "events": [], + "want_more": false, + "offset": 0, + "cursor": { + "last_timestamp": state.end_time + }, + } + )) +tags: +{{#if preserve_original_event}} + - preserve_original_event +{{/if}} +{{#if preserve_duplicate_custom_fields}} + - preserve_duplicate_custom_fields +{{/if}} +{{#each tags as |tag|}} + - {{tag}} +{{/each}} +{{#contains "forwarded" tags}} +publisher_pipeline.disable_host: true +{{/contains}} +{{#if processors}} +processors: +{{processors}} +{{/if}} \ No newline at end of file diff --git a/packages/sublime_security/data_stream/message_event/elasticsearch/ingest_pipeline/default.yml b/packages/sublime_security/data_stream/message_event/elasticsearch/ingest_pipeline/default.yml new file mode 100644 index 00000000000..78c906df39f --- /dev/null +++ b/packages/sublime_security/data_stream/message_event/elasticsearch/ingest_pipeline/default.yml @@ -0,0 +1,362 @@ +--- +description: Pipeline for processing message_event logs. +processors: + - set: + field: ecs.version + tag: set_ecs_version + value: 8.11.0 + - fail: + tag: data_collection_error + if: ctx.error?.message != null && ctx.message == null && ctx.event?.original == null + message: error message set and no data to process. + - rename: + field: message + tag: rename_message_to_event_original + target_field: event.original + ignore_missing: true + description: Renames the original `message` field to `event.original` to store a copy of the original message. The `event.original` field is not touched if the document already has one; it may happen when Logstash sends the document. + if: ctx.event?.original == null + - remove: + field: message + tag: remove_message + ignore_missing: true + description: The `message` field is no longer required if the document has an `event.original` field. + if: ctx.event?.original != null + - json: + field: event.original + tag: json_event_original + target_field: json + on_failure: + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - set: + field: event.kind + tag: set_event_kind_to_event + value: event + - append: + field: event.category + tag: append_email_into_event_category + value: email + - append: + field: event.type + tag: append_info_into_event_type + value: info + - set: + field: observer.vendor + tag: set_observer_vendor + value: Sublime Security + - set: + field: observer.product + tag: set_observer_product + value: Sublime Security + - rename: + field: json.data.message.canonical_id + tag: rename_data_message_canonicalid + target_field: sublime_security.message_event.canonical_id + ignore_missing: true + - rename: + field: json.canonical_id + tag: rename_canonical_id + target_field: sublime_security.message_event.canonical_id + ignore_missing: true + - date: + field: json.created_at + tag: date_created_at + target_field: sublime_security.message_event.created_at + formats: + - ISO8601 + if: ctx.json?.created_at != null && ctx.json.created_at != '' + on_failure: + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - set: + field: '@timestamp' + tag: set_@timestamp_from_message_event_created_at + copy_from: sublime_security.message_event.created_at + ignore_empty_value: true + - foreach: + field: json.data.flagged_rules + if: ctx.json?.data?.flagged_rules instanceof List + processor: + append: + field: rule.id + tag: append_data_flagged_rules_id_into_rule_id + value: '{{{_ingest._value.id}}}' + allow_duplicates: false + - foreach: + field: json.data.flagged_rules + if: ctx.json?.data?.flagged_rules instanceof List + processor: + append: + field: rule.name + tag: append_data_flagged_rules_name_into_rule_name + value: '{{{_ingest._value.name}}}' + allow_duplicates: false + - rename: + field: json.data.flagged_rules + tag: rename_data_flagged_rules + target_field: sublime_security.message_event.data.flagged_rules + ignore_missing: true + - foreach: + field: sublime_security.message_event.data.flagged_rules + tag: foreach_sublime_security_message_event_data_flagged_rules_to_remove_id_and_name + if: ctx.sublime_security?.message_event?.data?.flagged_rules instanceof List + processor: + remove: + field: + - _ingest._value.id + - _ingest._value.name + tag: remove_duplicate_custom_fields_from_data_flagged_rules_array + ignore_missing: true + if: ctx.tags == null || !(ctx.tags.contains('preserve_duplicate_custom_fields')) + - rename: + field: json.data.triggered_actions + tag: rename_data_triggered_actions + target_field: sublime_security.message_event.data.triggered_actions + ignore_missing: true + - rename: + field: json.data.message.external_id + tag: rename_data_message_external_id + target_field: sublime_security.message_event.external_id + ignore_missing: true + - rename: + field: json.external_id + tag: rename_external_id + target_field: sublime_security.message_event.external_id + ignore_missing: true + - rename: + field: json.forward_recipients + tag: rename_forward_recipients + target_field: sublime_security.message_event.forward_recipients + ignore_missing: true + - date: + field: json.forwarded_at + tag: date_forwarded_at + target_field: sublime_security.message_event.forwarded_at + formats: + - ISO8601 + if: ctx.json?.forwarded_at != null && ctx.json.forwarded_at != '' + on_failure: + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - rename: + field: json.data.message.id + tag: rename_data_message_id + target_field: sublime_security.message_event.id + ignore_missing: true + - rename: + field: json.id + tag: rename_id + target_field: sublime_security.message_event.id + ignore_missing: true + - set: + field: event.id + tag: set_event_id_from_message_event_id + copy_from: sublime_security.message_event.id + ignore_empty_value: true + - convert: + field: json.landed_in_spam + tag: convert_landed_in_spam_to_boolean + target_field: sublime_security.message_event.landed_in_spam + type: boolean + ignore_missing: true + on_failure: + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - rename: + field: json.mailbox.email + tag: rename_mailbox_email + target_field: sublime_security.message_event.mailbox.email + ignore_missing: true + - rename: + field: json.data.message.mailbox.external_id + tag: rename_data_message_mailbox_external_id + target_field: sublime_security.message_event.mailbox.external_id + ignore_missing: true + - rename: + field: json.mailbox.external_id + tag: rename_mailbox_external_id + target_field: sublime_security.message_event.mailbox.external_id + ignore_missing: true + - rename: + field: json.data.message.mailbox.id + tag: rename_data_message_mailbox_id + target_field: sublime_security.message_event.mailbox.id + ignore_missing: true + - rename: + field: json.mailbox.id + tag: rename_mailbox_id + target_field: sublime_security.message_event.mailbox.id + ignore_missing: true + - rename: + field: json.data.message.message_source_id + tag: rename_data_message_message_source_id + target_field: sublime_security.message_event.message_source_id + ignore_missing: true + - rename: + field: json.message_source_id + tag: rename_message_source_id + target_field: sublime_security.message_event.message_source_id + ignore_missing: true + - date: + field: json.read_at + tag: date_read_at + target_field: sublime_security.message_event.read_at + formats: + - ISO8601 + if: ctx.json?.read_at != null && ctx.json.read_at != '' + on_failure: + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - foreach: + field: json.recipients + if: ctx.json?.recipients instanceof List + processor: + append: + field: email.to.address + tag: append_recipients_email_into_email_to_address + value: '{{{_ingest._value.email}}}' + allow_duplicates: false + - foreach: + field: json.recipients + if: ctx.json?.recipients instanceof List + processor: + append: + field: related.user + tag: append_recipients_email_into_related.user + value: '{{{_ingest._value.email}}}' + allow_duplicates: false + - rename: + field: json.recipients + tag: rename_recipients + target_field: sublime_security.message_event.recipients + ignore_missing: true + - date: + field: json.replied_at + tag: date_replied_at + target_field: sublime_security.message_event.replied_at + formats: + - ISO8601 + if: ctx.json?.replied_at != null && ctx.json.replied_at != '' + on_failure: + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - rename: + field: json.sender.display_name + tag: rename_sender_display_name + target_field: sublime_security.message_event.sender.display_name + ignore_missing: true + - set: + field: source.user.name + tag: set_source_user_name_from_message_event_sender_display_name + copy_from: sublime_security.message_event.sender.display_name + ignore_empty_value: true + - append: + field: related.user + tag: append_sublime_security_message_event_sender_display_name_into_related_user + value: '{{{sublime_security.message_event.sender.display_name}}}' + allow_duplicates: false + if: ctx.sublime_security?.message_event?.sender?.display_name != null + - rename: + field: json.sender.email + tag: rename_sender_email + target_field: sublime_security.message_event.sender.email + ignore_missing: true + - append: + field: email.from.address + tag: append_sublime_security_message_event_sender_email_into_email_from_address + value: '{{{sublime_security.message_event.sender.email}}}' + allow_duplicates: false + if: ctx.sublime_security?.message_event?.sender?.email != null + - append: + field: related.user + tag: append_sublime_security_message_event_sender_email_into_related_user + value: '{{{sublime_security.message_event.sender.email}}}' + allow_duplicates: false + if: ctx.sublime_security?.message_event?.sender?.email != null + - rename: + field: json.subject + tag: rename_subject + target_field: sublime_security.message_event.subject + ignore_missing: true + - set: + field: email.subject + tag: set_email_subject_from_message_event_subject + copy_from: sublime_security.message_event.subject + ignore_empty_value: true + - rename: + field: json.type + tag: rename_type + target_field: sublime_security.message_event.type + ignore_missing: true + - foreach: + field: sublime_security.message_event.data.flagged_rules + if: ctx.sublime_security?.message_event?.data?.flagged_rules instanceof List + processor: + remove: + field: + - _ingest._value.id + - _ingest._value.name + tag: remove_custom_duplicate_fields_from_sublime_security_message_event_data_flagged_rules + ignore_missing: true + if: ctx.tags == null || !(ctx.tags.contains('preserve_duplicate_custom_fields')) + - foreach: + field: sublime_security.message_event.recipients + if: ctx.sublime_security?.message_event?.recipients instanceof List + processor: + remove: + field: _ingest._value.email + tag: remove_custom_duplicate_fields_from_sublime_security_message_event_recipients + ignore_missing: true + if: ctx.tags == null || !(ctx.tags.contains('preserve_duplicate_custom_fields')) + - remove: + field: + - sublime_security.message_event.created_at + - sublime_security.message_event.id + - sublime_security.message_event.sender.display_name + - sublime_security.message_event.sender.email + - sublime_security.message_event.subject + tag: remove_custom_duplicate_fields + ignore_missing: true + if: ctx.tags == null || !(ctx.tags.contains('preserve_duplicate_custom_fields')) + - remove: + field: json + tag: remove_json + ignore_missing: true + - script: + lang: painless + description: Drops null/empty values recursively. + tag: painless_remove_null + source: |- + boolean drop(Object object) { + if (object == null || object == '') { + return true; + } else if (object instanceof Map) { + ((Map) object).values().removeIf(v -> drop(v)); + return (((Map) object).size() == 0); + } else if (object instanceof List) { + ((List) object).removeIf(v -> drop(v)); + return (((List) object).length == 0); + } + return false; + } + drop(ctx); + - set: + field: event.kind + value: pipeline_error + tag: set_pipeline_error_into_event_kind + if: ctx.error?.message != null +on_failure: + - append: + field: error.message + value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}' + - set: + field: event.kind + tag: set_pipeline_error_to_event_kind + value: pipeline_error diff --git a/packages/sublime_security/data_stream/message_event/fields/base-fields.yml b/packages/sublime_security/data_stream/message_event/fields/base-fields.yml new file mode 100644 index 00000000000..99f454dd95b --- /dev/null +++ b/packages/sublime_security/data_stream/message_event/fields/base-fields.yml @@ -0,0 +1,20 @@ +- name: data_stream.type + type: constant_keyword + description: Data stream type. +- name: data_stream.dataset + type: constant_keyword + description: Data stream dataset. +- name: data_stream.namespace + type: constant_keyword + description: Data stream namespace. +- name: event.module + type: constant_keyword + description: Event module. + value: sublime_security +- name: event.dataset + type: constant_keyword + description: Event dataset. + value: sublime_security.message_event +- name: '@timestamp' + type: date + description: Event timestamp. diff --git a/packages/sublime_security/data_stream/message_event/fields/beats.yml b/packages/sublime_security/data_stream/message_event/fields/beats.yml new file mode 100644 index 00000000000..4084f1dc7f5 --- /dev/null +++ b/packages/sublime_security/data_stream/message_event/fields/beats.yml @@ -0,0 +1,6 @@ +- name: input.type + type: keyword + description: Type of filebeat input. +- name: log.offset + type: long + description: Log offset. diff --git a/packages/sublime_security/data_stream/message_event/fields/fields.yml b/packages/sublime_security/data_stream/message_event/fields/fields.yml new file mode 100644 index 00000000000..842ba38b1d8 --- /dev/null +++ b/packages/sublime_security/data_stream/message_event/fields/fields.yml @@ -0,0 +1,95 @@ +- name: sublime_security + type: group + fields: + - name: message_event + type: group + fields: + - name: canonical_id + type: keyword + description: Canonical ID of the message. + - name: created_at + type: date + description: Time this message was added to sublime_security. + - name: data + type: group + fields: + - name: flagged_rules + type: group + fields: + - name: id + type: keyword + description: ID of the flagged rule. + - name: name + type: keyword + description: Name of the flagged rule. + - name: severity + type: keyword + description: Severity of the flagged rule. + - name: tags + type: keyword + description: List of tags for the flagged rule. + - name: triggered_actions + type: group + fields: + - name: id + type: keyword + - name: name + type: keyword + - name: type + type: keyword + - name: external_id + type: keyword + description: ID of the message in the source system (e.g., Office 365 or Google Workspace). + - name: forward_recipients + type: keyword + description: Email addresses this message was forwarded to by the recipient mailbox. + - name: forwarded_at + type: date + description: Time this message was forwarded by the recipient mailbox. A null value indicates that it has not yet been forwarded. + - name: id + type: keyword + description: Message ID. + - name: landed_in_spam + type: boolean + description: Whether the message landed in the recipient's spam folder. + - name: mailbox + type: group + fields: + - name: email + type: keyword + description: Mailbox email address. + - name: external_id + type: keyword + description: ID of the mailbox in the source system (e.g., Office 365 or Google Workspace). + - name: id + type: keyword + description: Mailbox ID. + - name: message_source_id + type: keyword + description: ID of the message source of the message. + - name: read_at + type: date + description: Time this message was read in the user's mailbox. A null value indicates that it has not yet been marked read. + - name: recipients + type: group + fields: + - name: email + type: keyword + description: Email address. + - name: replied_at + type: date + description: Time that this message was replied to by the recipient mailbox. A null value indicates that it has not yet been replied to by the recipient. + - name: sender + type: group + fields: + - name: display_name + type: keyword + description: Display name. + - name: email + type: keyword + description: Email address. + - name: subject + type: keyword + description: Subject of the message. + - name: type + type: keyword diff --git a/packages/sublime_security/data_stream/message_event/manifest.yml b/packages/sublime_security/data_stream/message_event/manifest.yml new file mode 100644 index 00000000000..316eff85db9 --- /dev/null +++ b/packages/sublime_security/data_stream/message_event/manifest.yml @@ -0,0 +1,196 @@ +title: Sublime Security Message Event logs +type: logs +streams: + - input: cel + title: Sublime Security Message Event logs + description: Collecting Sublime Security Message Event logs via API. + enabled: false + template_path: cel.yml.hbs + vars: + - name: initial_interval + type: text + title: Initial Interval + multi: false + required: true + show_user: true + default: 24h + description: How far back to pull the Message Event logs from Sublime Security API. Supported units for this parameter are h/m/s. + - name: interval + type: text + title: Interval + description: Duration between requests to the Sublime Security API. Supported units for this parameter are h/m/s. + default: 1h + multi: false + required: true + show_user: true + - name: page_size + type: text + title: Page Size + multi: false + required: true + show_user: false + description: Page size for the response of the Sublime Security API. + default: 500 + - name: http_client_timeout + type: text + title: HTTP Client Timeout + description: Duration before declaring that the HTTP client connection has timed out. Supported time units are ns, us, ms, s, m, h. + multi: false + required: true + show_user: false + default: 30s + - name: enable_request_tracer + type: bool + title: Enable request tracing + multi: false + required: false + show_user: false + description: >- + The request tracer logs requests and responses to the agent's local file-system for debugging configurations. Enabling this request tracing compromises security and should only be used for debugging. See [documentation](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-cel.html#_resource_tracer_filename) for details. + - name: tags + type: text + title: Tags + multi: true + required: true + show_user: false + default: + - forwarded + - sublime_security-message_event + - name: preserve_original_event + required: false + show_user: true + title: Preserve original event + description: Preserves a raw copy of the original event, added to the field `event.original`. + type: bool + multi: false + default: false + - name: preserve_duplicate_custom_fields + required: false + show_user: false + title: Preserve duplicate custom fields + description: Preserve sublime_security.message_event fields that were copied to Elastic Common Schema (ECS) fields. + type: bool + multi: false + default: false + - name: processors + type: yaml + title: Processors + multi: false + required: false + show_user: false + description: >- + Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. + - input: aws-s3 + template_path: aws-s3.yml.hbs + title: Sublime Security Message Event logs via AWS S3 or SQS + description: Collecting Sublime Security Message Event logs via AWS S3 or SQS input. + enabled: false + vars: + - name: bucket_arn + type: text + title: '[S3] Bucket ARN' + multi: false + required: false + show_user: true + description: ARN of the AWS S3 bucket that will be polled for list operation. It is a required parameter for collecting logs via the AWS S3. + - name: bucket_list_prefix + type: text + title: '[S3] Bucket Prefix' + multi: false + required: false + show_user: true + description: Prefix to apply for the list request to the S3 bucket. + - name: interval + type: text + title: '[S3] Interval' + multi: false + required: false + show_user: true + default: 120s + description: Listing of the S3 bucket will be polled according to the time interval defined by bucket_list_interval config. Default value is 120 secs. Supported units for this parameter are h/m/s. + - name: number_of_workers + type: integer + title: '[S3] Number of Workers' + multi: false + required: false + show_user: true + default: 5 + description: Number of workers that will process the S3 objects listed. + - name: queue_url + type: text + title: '[SQS] Queue URL' + multi: false + required: false + show_user: true + description: URL of the AWS SQS queue that messages will be received from. It is a required parameter for collecting logs via the AWS SQS. + - name: visibility_timeout + type: text + title: '[SQS] Visibility Timeout' + multi: false + required: false + show_user: true + default: 300s + description: The duration that the received messages are hidden from subsequent retrieve requests after being retrieved by a ReceiveMessage request. The maximum is 12 hours. Supported units for this parameter are h/m/s. + - name: api_timeout + type: text + title: '[SQS] API Timeout' + multi: false + required: false + show_user: true + default: 120s + description: The maximum duration of AWS API can take. The maximum is half of the visibility timeout value. Supported units for this parameter are h/m/s. + - name: max_number_of_messages + type: integer + title: '[SQS] Maximum Concurrent SQS Messages' + required: false + show_user: true + default: 5 + description: The maximum number of SQS messages that can be inflight at any time. + - name: file_selectors + type: yaml + title: '[SQS] File Selectors' + multi: false + required: false + show_user: false + description: >- + If the SQS queue will have events that correspond to files that this integration shouldn’t process, file_selectors can be used to limit the files that are downloaded. This is a list of selectors which are made up of regex and expand_event_list_from_field options. The regex should match the S3 object key in the SQS message, and the optional expand_event_list_from_field is the same as the global setting. If file_selectors is given, then any global expand_event_list_from_field value is ignored in favor of the ones specified in the file_selectors. Regexes use [RE2 syntax](https://pkg.go.dev/regexp/syntax). Files that do not match one of the regexes will not be processed. + - name: external_id + type: text + title: External ID + multi: false + required: false + show_user: false + description: External ID to use when assuming a role in another account. + - name: tags + type: text + title: Tags + multi: true + required: true + show_user: false + default: + - forwarded + - sublime_security-message_event + - name: preserve_original_event + required: false + show_user: true + title: Preserve original event + description: Preserves a raw copy of the original event, added to the field `event.original`. + type: bool + multi: false + default: false + - name: preserve_duplicate_custom_fields + required: true + show_user: false + title: Preserve duplicate custom fields + description: Preserve sublime_security.message_event fields that were mapped to Elastic Common Schema (ECS) fields. + type: bool + multi: false + default: false + - name: processors + type: yaml + title: Processors + multi: false + required: false + show_user: false + description: >- + Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See [Processors](https://www.elastic.co/guide/en/beats/filebeat/current/filtering-and-enhancing-data.html) for details. diff --git a/packages/sublime_security/data_stream/message_event/sample_event.json b/packages/sublime_security/data_stream/message_event/sample_event.json new file mode 100644 index 00000000000..8e2abc970d2 --- /dev/null +++ b/packages/sublime_security/data_stream/message_event/sample_event.json @@ -0,0 +1,114 @@ +{ + "@timestamp": "2024-07-12T05:15:08.221Z", + "agent": { + "ephemeral_id": "384edc61-b94b-40cf-9cc6-86d5418d35e5", + "id": "5f3fcbb9-1a97-4ff3-857f-167af6664464", + "name": "docker-fleet-agent", + "type": "filebeat", + "version": "8.13.0" + }, + "data_stream": { + "dataset": "sublime_security.message_event", + "namespace": "17638", + "type": "logs" + }, + "ecs": { + "version": "8.11.0" + }, + "elastic_agent": { + "id": "5f3fcbb9-1a97-4ff3-857f-167af6664464", + "snapshot": false, + "version": "8.13.0" + }, + "email": { + "from": { + "address": [ + "bob.demo@gmail.com" + ] + }, + "subject": "Urgent: Wire transfer", + "to": { + "address": [ + "xyz@example.com", + "user12@example.com", + "user@example.com", + "leon12@example.com" + ] + } + }, + "event": { + "agent_id_status": "verified", + "category": [ + "email" + ], + "dataset": "sublime_security.message_event", + "id": "9c426680-5cdf-4283-adbd-d79ba0e52434", + "ingested": "2024-08-28T10:36:57Z", + "kind": "event", + "original": "{\"canonical_id\":\"dd97dc82731ff7e82edfccaef59826cccd271bd4423e09d1e150ade83037cb37\",\"created_at\":\"2024-07-12T05:15:08.221838Z\",\"external_id\":\"7a2dfbeb-1310-48fc-9ed9-f480608a0306\",\"forward_recipients\":[],\"forwarded_at\":null,\"id\":\"9c426680-5cdf-4283-adbd-d79ba0e52434\",\"landed_in_spam\":false,\"mailbox\":{\"email\":\"demo@example.com\",\"external_id\":null,\"id\":\"433fe142-e2e5-4372-84ea-480279543a9b\"},\"message_source_id\":\"257982a1-f106-4c68-bc64-ff032914ed5f\",\"read_at\":null,\"recipients\":[{\"email\":\"xyz@example.com\"},{\"email\":\"user12@example.com\"},{\"email\":\"user@example.com\"},{\"email\":\"leon12@example.com\"}],\"replied_at\":null,\"sender\":{\"display_name\":\"Bob Doe\",\"email\":\"bob.demo@gmail.com\"},\"subject\":\"Urgent: Wire transfer\"}", + "type": [ + "info" + ] + }, + "input": { + "type": "cel" + }, + "observer": { + "product": "Sublime Security", + "vendor": "Sublime Security" + }, + "related": { + "user": [ + "xyz@example.com", + "user12@example.com", + "user@example.com", + "leon12@example.com", + "Bob Doe", + "bob.demo@gmail.com" + ] + }, + "source": { + "user": { + "name": "Bob Doe" + } + }, + "sublime_security": { + "message_event": { + "canonical_id": "dd97dc82731ff7e82edfccaef59826cccd271bd4423e09d1e150ade83037cb37", + "created_at": "2024-07-12T05:15:08.221Z", + "external_id": "7a2dfbeb-1310-48fc-9ed9-f480608a0306", + "id": "9c426680-5cdf-4283-adbd-d79ba0e52434", + "landed_in_spam": false, + "mailbox": { + "email": "demo@example.com", + "id": "433fe142-e2e5-4372-84ea-480279543a9b" + }, + "message_source_id": "257982a1-f106-4c68-bc64-ff032914ed5f", + "recipients": [ + { + "email": "xyz@example.com" + }, + { + "email": "user12@example.com" + }, + { + "email": "user@example.com" + }, + { + "email": "leon12@example.com" + } + ], + "sender": { + "display_name": "Bob Doe", + "email": "bob.demo@gmail.com" + }, + "subject": "Urgent: Wire transfer" + } + }, + "tags": [ + "preserve_original_event", + "preserve_duplicate_custom_fields", + "forwarded", + "sublime_security-message_event" + ] +} \ No newline at end of file diff --git a/packages/sublime_security/docs/README.md b/packages/sublime_security/docs/README.md new file mode 100644 index 00000000000..509daaa6bdf --- /dev/null +++ b/packages/sublime_security/docs/README.md @@ -0,0 +1,1537 @@ +# Sublime Security + +Sublime Security is a programmable, AI-powered, cloud email security platform for Microsoft 365 and Google Workspace environments. It is used to block email attacks such as phishing, BEC, malware, threat hunt, and auto-triage user reports. + +The Sublime Security integration collects data for Audit, Email Message(MDM Schema) and Message Event logs using REST API and AWS-S3 or AWS-SQS: + +- REST API mode - Sublime Security integration collects and parses data from the Sublime Security REST APIs. +- AWS S3 polling mode - Sublime Security writes data to S3 and Elastic Agent polls the S3 bucket by listing its contents and reading new files. +- AWS S3 SQS mode - Sublime Security writes data to S3, S3 pushes a new object notification to SQS, Elastic Agent receives the notification from SQS, and then reads the S3 object. Multiple Agents can be used in this mode. + +## Data streams + +The Sublime Security integration collects three types of logs: + +**[Audit](https://docs.sublime.security/reference/listeventsinauditlog)** - Captures detailed records of all significant actions and changes within the platform, including changes to email security policies, user access to email data, and modifications to email configurations, ensuring traceability and compliance for all operations. + +**[Email Message](https://docs.sublime.security/docs/export-message-mdms)** - Represents the flow of individual emails through the platform, including sender and recipient details, spam filtering outcomes, and overall email disposition, helping to secure and analyze email communication. + +**[Message Event](https://docs.sublime.security/reference/getmessage-1)** - Represents document specific actions taken on emails, like spam detection or rule applications, providing detailed insights into how the platform processes and protects email communications. + +## Requirements + +Elastic Agent must be installed. For more information, refer to the link [here](https://www.elastic.co/guide/en/fleet/current/elastic-agent-installation.html). + +### Installing and managing an Elastic Agent: + +You have a few options for installing and managing an Elastic Agent: + +### Install a Fleet-managed Elastic Agent (recommended): + +With this approach, you install Elastic Agent and use Fleet in Kibana to define, configure, and manage your agents in a central location. We recommend using Fleet management because it makes the management and upgrade of your agents considerably easier. + +### Install Elastic Agent in standalone mode (advanced users): + +With this approach, you install Elastic Agent and manually configure the agent locally on the system where it’s installed. You are responsible for managing and upgrading the agents. This approach is reserved for advanced users only. + +### Install Elastic Agent in a containerized environment: + +You can run Elastic Agent inside a container, either with Fleet Server or standalone. Docker images for all versions of Elastic Agent are available from the Elastic Docker registry, and we provide deployment manifests for running on Kubernetes. + +There are some minimum requirements for running Elastic Agent and for more information, refer to the link [here](https://www.elastic.co/guide/en/fleet/current/elastic-agent-installation.html#_minimum_requirements). + +## Setup + +### To collect data from the Sublime Security API: + +#### Step 1: Go to Platform +- Visit the [Sublime Security Platform](https://platform.sublime.security/) and select `API` in Developers section. + +#### Step 2: Generating the API Key +- Retrieve your `API Key`. This key will be used further in the Elastic integration setup to authenticate and access different Sublime Security Logs. +- `Base URL` of Sublime Security is also required for configuring integration. + +**Note**: Users with the `Admin` role are allowed to access `Audit` logs. For more information, refer [here](https://docs.sublime.security/docs/role-based-access-control-rbac). + +### To collect data from AWS S3 Bucket or AWS SQS: + +#### For AWS S3 Bucket, follow the below steps: +- Create an Amazon S3 bucket. Refer to the link [here](https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html). +- User can set the parameter "Bucket List Prefix" according to the requirement. + +#### For AWS SQS, follow the below steps: +1. If data forwarding to an AWS S3 Bucket hasn't been configured, then first set up an AWS S3 Bucket as mentioned in the above documentation. +2. To set up an SQS queue, follow "Step 1: Create an Amazon SQS queue" mentioned in the [Documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/ways-to-add-notification-config-to-bucket.html). + - While creating an SQS Queue, please provide the same bucket ARN that has been generated after creating an AWS S3 Bucket. +3. Set up event notifications for a S3 bucket. Follow this [link](https://docs.aws.amazon.com/AmazonS3/latest/userguide/enable-event-notifications.html). + - Users have to set the prefix parameter the same as the S3 Bucket List Prefix as created earlier. (for example, `exports/sublime_platform_audit_log/` for a audit data stream). + - Select the event type as s3:ObjectCreated:*, select the destination type SQS Queue, and select the queue that has been created in Step 2. + +**Note**: + - Credentials for the above AWS S3 and SQS input types should be configured using the [link](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-aws-s3.html#aws-credentials-config). + - Data collection via AWS S3 Bucket and AWS SQS are mutually exclusive in this case. + - You can configure a global SQS queue for all data streams or a local SQS queue for each data stream. Configuring data stream specific SQS queues will enable better performance and scalability. Data stream specific SQS queues will always override any global queue definitions for that specific data stream. + +### Enabling the integration in Elastic: + +1. In Kibana go to Management > Integrations. +2. In "Search for integrations" search bar, type Sublime Security. +3. Click on the "Sublime Security" integration from the search results. +4. Click on the "Add Sublime Security" button to add the integration. +5. Enable the Integration to collect logs via AWS S3 or API input. +6. Under the AWS S3 input, there are two types of inputs: using AWS S3 Bucket or using SQS. +7. Add all the required integration configuration parameters, including API Key, Interval, Initial Interval and Page Size for API input and Access Key, Secret Key and Session Token for AWS input type to enable data collection. +8. Click on "Save and continue" to save the integration. + +**Note**: +- The Base URL for Sublime Security cloud customers is `https://api.platform.sublimesecurity.com`. Depending on your type of deployment, yours may be different. +- For SSO users, in addition to access key ID and secret access key, the session token is required to configure integration. For IAM users, the session token is optional and not required. + +## Logs reference + +### Audit + +This is the `audit` dataset. + +#### Example + +An example event for `audit` looks as following: + +```json +{ + "@timestamp": "2024-08-12T06:04:03.714Z", + "agent": { + "ephemeral_id": "390c3f2d-c9eb-4229-9992-0f4fc2436f51", + "id": "5f3fcbb9-1a97-4ff3-857f-167af6664464", + "name": "docker-fleet-agent", + "type": "filebeat", + "version": "8.13.0" + }, + "data_stream": { + "dataset": "sublime_security.audit", + "namespace": "99243", + "type": "logs" + }, + "ecs": { + "version": "8.11.0" + }, + "elastic_agent": { + "id": "5f3fcbb9-1a97-4ff3-857f-167af6664464", + "snapshot": false, + "version": "8.13.0" + }, + "event": { + "action": "search", + "agent_id_status": "verified", + "dataset": "sublime_security.audit", + "id": "bd49af79-0cfb-4184-bd18-b0401d69ac61", + "ingested": "2024-08-28T10:35:52Z", + "kind": "event", + "original": "{\"created_at\":\"2024-08-12T06:04:03.714126Z\",\"created_by\":{\"active\":true,\"created_at\":\"2024-07-12T05:13:47.879426Z\",\"email_address\":\"demo@example.com\",\"first_name\":\"Demo\",\"google_oauth_user_id\":\"d83rb8et4-refe-fe7t4f8efe\",\"id\":\"6e6eca05-4fea-406b-86d4-b40177e25474\",\"is_enrolled\":true,\"last_name\":\"User\",\"microsoft_oauth_user_id\":\"fhe7t4bgf8-freu-ebfur94ref\",\"phone_number\":null,\"role\":\"admin\",\"updated_at\":\"2024-07-12T05:13:47.879426Z\"},\"data\":{\"request\":{\"api_key_name\":\"demo mode local\",\"authentication_method\":\"api_key\",\"body\":\"\",\"id\":\"6ad202de-0def-423d-a0f2-549402e1a9c9\",\"ip\":\"1.128.0.0\",\"method\":\"GET\",\"path\":\"/v0/message-groups\",\"user_agent\":\"Go-http-client/1.1\"}},\"id\":\"bd49af79-0cfb-4184-bd18-b0401d69ac61\",\"type\":\"message_group.search\"}", + "type": [ + "info" + ] + }, + "http": { + "request": { + "id": "6ad202de-0def-423d-a0f2-549402e1a9c9", + "method": "GET" + } + }, + "input": { + "type": "cel" + }, + "observer": { + "product": "Sublime Security", + "vendor": "Sublime Security" + }, + "related": { + "ip": [ + "1.128.0.0" + ], + "user": [ + "demo@example.com", + "Demo", + "d83rb8et4-refe-fe7t4f8efe", + "6e6eca05-4fea-406b-86d4-b40177e25474", + "fhe7t4bgf8-freu-ebfur94ref" + ] + }, + "source": { + "ip": "1.128.0.0" + }, + "sublime_security": { + "audit": { + "created_at": "2024-08-12T06:04:03.714Z", + "created_by": { + "active": true, + "created_at": "2024-07-12T05:13:47.879Z", + "email_address": "demo@example.com", + "first_name": "Demo", + "google_oauth_user_id": "d83rb8et4-refe-fe7t4f8efe", + "id": "6e6eca05-4fea-406b-86d4-b40177e25474", + "is_enrolled": true, + "last_name": "User", + "microsoft_oauth_user_id": "fhe7t4bgf8-freu-ebfur94ref", + "role": "admin", + "updated_at": "2024-07-12T05:13:47.879Z" + }, + "data": { + "request": { + "api_key_name": "demo mode local", + "authentication_method": "api_key", + "id": "6ad202de-0def-423d-a0f2-549402e1a9c9", + "ip": "1.128.0.0", + "method": "GET", + "path": "/v0/message-groups", + "user_agent": "Go-http-client/1.1" + } + }, + "id": "bd49af79-0cfb-4184-bd18-b0401d69ac61", + "type": "message_group.search" + } + }, + "tags": [ + "preserve_original_event", + "preserve_duplicate_custom_fields", + "forwarded", + "sublime_security-audit" + ], + "url": { + "path": "/v0/message-groups" + }, + "user": { + "domain": "example.com", + "email": "demo@example.com", + "full_name": "Demo User", + "id": "6e6eca05-4fea-406b-86d4-b40177e25474", + "name": "demo", + "roles": [ + "admin" + ] + }, + "user_agent": { + "device": { + "name": "Other" + }, + "name": "Go-http-client", + "original": "Go-http-client/1.1", + "version": "1.1" + } +} +``` + +**Exported fields** + +| Field | Description | Type | +|---|---|---| +| @timestamp | Event timestamp. | date | +| data_stream.dataset | Data stream dataset. | constant_keyword | +| data_stream.namespace | Data stream namespace. | constant_keyword | +| data_stream.type | Data stream type. | constant_keyword | +| event.dataset | Event dataset. | constant_keyword | +| event.module | Event module. | constant_keyword | +| input.type | Type of filebeat input. | keyword | +| log.offset | Log offset. | long | +| sublime_security.audit.created_at | Event creation time. | date | +| sublime_security.audit.created_by.active | | boolean | +| sublime_security.audit.created_by.created_at | User creation time. | date | +| sublime_security.audit.created_by.deleted_at | User deletion time. | date | +| sublime_security.audit.created_by.email_address | Email address. | keyword | +| sublime_security.audit.created_by.first_name | First name. | keyword | +| sublime_security.audit.created_by.google_oauth_user_id | The user's Google user ID, if it exists. | keyword | +| sublime_security.audit.created_by.id | User ID. | keyword | +| sublime_security.audit.created_by.is_enrolled | Whether the user has begun using the system (e.g. accepted an invitation or logged in at least once). | boolean | +| sublime_security.audit.created_by.last_name | Last name. | keyword | +| sublime_security.audit.created_by.microsoft_oauth_user_id | The user's Microsoft user ID, if it exists. | keyword | +| sublime_security.audit.created_by.phone_number | Phone number. | keyword | +| sublime_security.audit.created_by.role | Role assumed by the user. | keyword | +| sublime_security.audit.created_by.updated_at | User last updated time. | date | +| sublime_security.audit.data.message.id | Message ID. | keyword | +| sublime_security.audit.data.message_group.id | Message Group ID. | keyword | +| sublime_security.audit.data.request.api_key_name | Name of API key if an API key was used. | keyword | +| sublime_security.audit.data.request.authentication_method | Description of how request was authenticated. | keyword | +| sublime_security.audit.data.request.body | Request body. | keyword | +| sublime_security.audit.data.request.id | API request ID. | keyword | +| sublime_security.audit.data.request.ip | IP address of requester, if available. | ip | +| sublime_security.audit.data.request.method | HTTP method. | keyword | +| sublime_security.audit.data.request.path | URL path. | keyword | +| sublime_security.audit.data.request.query | Query parameters. | object | +| sublime_security.audit.data.request.user_agent | User agent of requester, if available. | keyword | +| sublime_security.audit.id | Event ID. | keyword | +| sublime_security.audit.type | Event type. | keyword | + + +### Email Message + +This is the `email_message` dataset. + +#### Example + +An example event for `email_message` looks as following: + +```json +{ + "@timestamp": "2024-08-02T07:40:25.135Z", + "agent": { + "ephemeral_id": "832ebf28-565e-4f38-a67e-ee5ea9f51e89", + "id": "5f3fcbb9-1a97-4ff3-857f-167af6664464", + "name": "docker-fleet-agent", + "type": "filebeat", + "version": "8.13.0" + }, + "aws": { + "s3": { + "bucket": { + "arn": "arn:aws:s3:::elastic-package-sublime-security-bucket-33881", + "name": "elastic-package-sublime-security-bucket-33881" + }, + "object": { + "key": "email-message.log" + } + } + }, + "cloud": { + "region": "us-east-1" + }, + "data_stream": { + "dataset": "sublime_security.email_message", + "namespace": "57022", + "type": "logs" + }, + "destination": { + "domain": "example.com", + "subdomain": "example", + "top_level_domain": "com" + }, + "ecs": { + "version": "8.11.0" + }, + "elastic_agent": { + "id": "5f3fcbb9-1a97-4ff3-857f-167af6664464", + "snapshot": false, + "version": "8.13.0" + }, + "email": { + "attachments": [ + { + "file": { + "extension": "pdf", + "hash": { + "md5": "1a2b3c", + "sha1": "4d5e6f", + "sha256": "7g8h9i" + }, + "mime_type": "application/pdf", + "name": "sample_document.pdf", + "size": 102400 + } + }, + { + "file": { + "extension": "jpg", + "hash": { + "md5": "7h8i9j", + "sha1": "1k2l3m", + "sha256": "4n5o6p" + }, + "mime_type": "image/jpeg", + "name": "image_photo.jpg", + "size": 204800 + } + }, + { + "file": { + "extension": "txt", + "hash": { + "md5": "1x2y3z", + "sha1": "4a5b6c", + "sha256": "7d8e9f" + }, + "mime_type": "text/plain", + "name": "notes.txt", + "size": 5120 + } + } + ], + "bcc": { + "address": [ + "john.doe@example.com" + ] + }, + "cc": { + "address": [ + "jane.smith@example.org" + ] + }, + "direction": "outbound", + "from": { + "address": [ + "testing@sublimesecurity.com" + ] + }, + "message_id": "2fe271830bbad5fe3a70abbe7a8c0bfe7refe3ffe", + "origination_timestamp": "2024-08-02T07:40:25.135Z", + "reply_to": { + "address": [ + "user@example.com" + ] + }, + "subject": "Sublime-Security-Standard-Test-String", + "to": { + "address": [ + "user@example.com" + ] + }, + "x_mailer": "MyCustomMailer" + }, + "event": { + "agent_id_status": "verified", + "category": [ + "email" + ], + "dataset": "sublime_security.email_message", + "id": "01911208-633c-7f03-b303-e594d92cf818", + "ingested": "2024-08-28T10:59:37Z", + "kind": "event", + "original": "{\"body\":{\"plain\":{\"raw\":\"Sublime Security test message.\\n\",\"charset\":\"utf-8\",\"content_transfer_encoding\":\"base64\"},\"current_thread\":{\"text\":\"Sublime Security test message.\"},\"html\":{\"charset\":\"utf-8\",\"content_transfer_encoding\":\"base64\",\"display_text\":\"Sublime Security test message.\",\"raw\":\"

Sublime Security test message.

\",\"inner_text\":\"

Sublime Security test message.

\"},\"ips\":[{\"ip\":\"1.128.0.0\"}],\"links\":[{\"display_text\":\"Click here!\",\"mismatched\":true,\"display_url\":{\"fragment\":\"search\",\"password\":\"pass123\",\"path\":\"/test\",\"port\":80,\"query_params\":\"q=elasticsearch\",\"rewrite\":{\"encoders\":[\"base64\"],\"original\":\"demo\"},\"scheme\":\"https\",\"url\":\"https://example.com/test?q=elasticsearch#search\",\"username\":\"test\",\"domain\":{\"domain\":\"example.com\",\"punycode\":\"demo\",\"root_domain\":\"example.com\",\"subdomain\":\"example\",\"tld\":\"com\",\"valid\":true,\"sld\":\"example\"}}},{\"href_url\":{\"fragment\":\"search\",\"password\":\"pass123\",\"path\":\"/test\",\"port\":80,\"query_params\":\"q=elasticsearch\",\"rewrite\":{\"encoders\":[\"base64\"],\"original\":\"demo\"},\"scheme\":\"https\",\"url\":\"https://example.com/test?q=elasticsearch#search\",\"username\":\"test\",\"domain\":{\"domain\":\"example.com\",\"punycode\":\"demo\",\"root_domain\":\"example.com\",\"subdomain\":\"example\",\"tld\":\"com\",\"valid\":true,\"sld\":\"example\"}}}]},\"external\":{\"created_at\":\"2024-08-02T07:40:25.135939305Z\",\"message_id\":\"2fe271830bbad5fe3a70abbe7a8c0bfe7refe3ffe\",\"route_type\":\"sent\",\"spam\":false,\"spam_folder\":true,\"thread_id\":\"sample_data\"},\"attachments\":[{\"content_id\":\"abc123\",\"content_transfer_encoding\":\"base64\",\"content_type\":\"application/pdf\",\"file_extension\":\".pdf\",\"file_name\":\"sample_document.pdf\",\"file_type\":\"document\",\"md5\":\"1a2b3c\",\"raw\":\"JVBERi0xLjMKJcfs4AAQSkZjRgABAQE\",\"sha1\":\"4d5e6f\",\"sha256\":\"7g8h9i\",\"size\":102400},{\"content_id\":\"xyz456\",\"content_transfer_encoding\":\"7bit\",\"content_type\":\"image/jpeg\",\"file_extension\":\".jpg\",\"file_name\":\"image_photo.jpg\",\"file_type\":\"image\",\"md5\":\"7h8i9j\",\"raw\":\"/9j/4AAQSkZJRgABAQEJVBERi0xLjMKJd\",\"sha1\":\"1k2l3m\",\"sha256\":\"4n5o6p\",\"size\":204800},{\"content_id\":\"efg789\",\"content_transfer_encoding\":\"quoted-printable\",\"content_type\":\"text/plain\",\"file_extension\":\".txt\",\"file_name\":\"notes.txt\",\"file_type\":\"text\",\"md5\":\"1x2y3z\",\"raw\":\"SGVsbG8gdVsbG8gd29yb29ybGQhVsbG8gd29yb\",\"sha1\":\"4a5b6c\",\"sha256\":\"7d8e9f\",\"size\":5120}],\"headers\":{\"x_authenticated_domain\":{\"domain\":\"example.com\",\"punycode\":\"xn--example-d4a.com\",\"root_domain\":\"example.com\",\"sld\":\"example\",\"subdomain\":\"sub\",\"tld\":\"com\",\"valid\":true},\"x_authenticated_sender\":{\"domain\":{\"domain\":\"example.com\",\"punycode\":\"example.com\",\"root_domain\":\"example.com\",\"sld\":\"example\",\"subdomain\":\"sub\",\"tld\":\"com\",\"valid\":true},\"email\":\"user@example.com\",\"local_part\":\"user\"},\"x_client_ip\":{\"ip\":\"1.128.0.0\"},\"x_originating_ip\":{\"ip\":\"1.128.0.0\"},\"x_secure_server_account\":\"account_value\",\"x_sender\":{\"domain\":{\"domain\":\"example.com\",\"punycode\":\"example.com\",\"root_domain\":\"example.com\",\"sld\":\"example\",\"subdomain\":\"sub\",\"tld\":\"com\",\"valid\":true},\"email\":\"user@example.com\",\"local_part\":\"user\"},\"return_path\":{\"domain\":{\"domain\":\"example.com\",\"punycode\":\"xn--example-d4a.com\",\"root_domain\":\"example\",\"sld\":\"example\",\"subdomain\":\"sub\",\"tld\":\"com\",\"valid\":true},\"email\":\"user@example.com\",\"local_part\":\"user\"},\"references\":[\"test1\",\"test2\"],\"auth_summary\":{\"dmarc\":{\"details\":{\"action\":\"quarantine\",\"disposition\":\"quarantine\",\"from\":{\"domain\":\"example.com\",\"punycode\":\"example.com\",\"root_domain\":\"example.com\",\"sld\":\"example\",\"subdomain\":\"example\",\"tld\":\"com\",\"valid\":true},\"policy\":\"reject\",\"sub_policy\":\"none\",\"verdict\":\"pass\",\"version\":\"1.0\"},\"pass\":true,\"received_hop\":1},\"spf\":{\"details\":{\"client_ip\":{\"ip\":\"1.128.0.0\"},\"description\":\"SPF record found\",\"designator\":\"pass\",\"helo\":{\"domain\":\"example.com\",\"punycode\":\"example.com\",\"root_domain\":\"example.com\",\"sld\":\"example\",\"subdomain\":\"example\",\"tld\":\"com\",\"valid\":true},\"server\":{\"domain\":\"mail.example.com\",\"punycode\":\"mail.example.com\",\"root_domain\":\"example.com\",\"sld\":\"example\",\"subdomain\":\"mail\",\"tld\":\"com\",\"valid\":true},\"verdict\":\"pass\"},\"error\":\"true\",\"pass\":true,\"received_hop\":2}},\"date\":\"2019-10-21T18:23:24Z\",\"date_original_offset\":\"-4\",\"hops\":[{\"index\":0,\"fields\":[{\"name\":\"To\",\"value\":\"user@example.com\",\"position\":0},{\"name\":\"Subject\",\"value\":\"Sublime-Security-Standard-Test-String\",\"position\":1},{\"name\":\"Date\",\"value\":\"Mon, 21 Oct 2019 14:23:24 -0400\",\"position\":2},{\"name\":\"From\",\"value\":\"Sublime Security Test \",\"position\":3}],\"authentication_results\":{\"compauth\":{\"verdict\":\"pass\",\"reason\":\"reason_value\"},\"dkim\":\"pass\",\"dkim_details\":{\"algorithm\":\"rsa-sha256\",\"body_hash\":\"abcdefg\",\"domain\":\"example.com\",\"headers\":\"from, to, subject\",\"instance\":\"example.com\",\"selector\":\"abcdefg\",\"signature\":\"abcdefg\",\"type\":\"dkim\",\"version\":\"1.0\"},\"dmarc\":\"pass\",\"dmarc_details\":{\"action\":\"quarantine\",\"disposition\":\"quarantine\",\"from\":{\"domain\":\"example.com\",\"punycode\":\"example.com\",\"root_domain\":\"example.com\",\"sld\":\"example\",\"subdomain\":\"example\",\"tld\":\"com\",\"valid\":true},\"policy\":\"reject\",\"sub_policy\":\"none\",\"verdict\":\"pass\",\"version\":\"1.0\"},\"instance\":\"example.com\",\"server\":{\"domain\":\"mail.example.com\",\"punycode\":\"mail.example.com\",\"root_domain\":\"example.com\",\"sld\":\"example\",\"subdomain\":\"mail\",\"tld\":\"com\",\"valid\":true},\"spf\":\"pass\",\"spf_details\":{\"client_ip\":{\"ip\":\"1.128.0.0\"},\"description\":\"SPF record found\",\"designator\":\"pass\",\"helo\":{\"domain\":\"example.com\",\"punycode\":\"example.com\",\"root_domain\":\"example.com\",\"sld\":\"example\",\"subdomain\":\"example\",\"tld\":\"com\",\"valid\":true},\"server\":{\"domain\":\"mail.example.com\",\"root_domain\":\"example.com\",\"sld\":\"example\",\"subdomain\":\"mail\",\"tld\":\"com\",\"valid\":true},\"verdict\":\"pass\"},\"type\":\"spf\"},\"received\":{\"additional\":{\"raw\":\"Authentication successful\"},\"id\":{\"raw\":\"msg-12345\"},\"link\":{\"raw\":\"https://mail.example.com/message/12345\"},\"mailbox\":{\"raw\":\"user@example.com\"},\"protocol\":{\"raw\":\"IMAP\"},\"server\":{\"raw\":\"imap.example.com\"},\"source\":{\"raw\":\"81.2.69.144\"},\"time\":\"2019-10-21T18:23:24Z\",\"zone_offset\":\"+00:00\"},\"received_spf\":{\"client_ip\":{\"ip\":\"1.128.0.0\"},\"description\":\"SPF record found\",\"designator\":\"pass\",\"helo\":{\"domain\":\"example.com\",\"punycode\":\"example.com\",\"root_domain\":\"example.com\",\"sld\":\"example\",\"subdomain\":\"example\",\"tld\":\"com\",\"valid\":true},\"server\":{\"domain\":\"mail.example.com\",\"punycode\":\"mail.example.com\",\"root_domain\":\"example.com\",\"sld\":\"example\",\"subdomain\":\"mail\",\"tld\":\"com\",\"valid\":true},\"verdict\":\"pass\"},\"signature\":{\"algorithm\":\"rsa-sha256\",\"body_hash\":\"b9c4a3f9d93d9a38bdf8c47a8f2d2c79ec1d8b1f\",\"domain\":\"example.com\",\"headers\":\"from:to:subject:date\",\"instance\":\"123456\",\"selector\":\"default\",\"signature\":\"d2abf9d6c8f4b8d68d8f3f7b6f9d3b8e6a8c2b3a9f4b8d7b9d3b6a8f9c3b4e5f\",\"type\":\"spf\",\"version\":\"1\"}}],\"in_reply_to\":\"in_reply_to_value\",\"delivered_to\":{\"domain\":{\"domain\":\"example.com\",\"subdomain\":\"example\",\"tld\":\"com\",\"email\":\"testing@sublimesecurity.com\",\"punycode\":\"example.com\",\"root_domain\":\"example.com\",\"sld\":\"example\",\"valid\":true},\"email\":\"testing@sublimesecurity.com\",\"local_part\":\"testing\"},\"ips\":[{\"ip\":\"1.128.0.0\"}],\"mailer\":\"MyCustomMailer\",\"message_id\":\"2fe271830bbad5fe3a70abbe7a8c0bfe7refe3ffe\",\"domains\":[{\"domain\":\"test.com\",\"subdomain\":\"test\",\"tld\":\"com\",\"punycode\":\"test.com\",\"root_domain\":\"test.com\",\"sld\":\"test\",\"valid\":true},{\"domain\":\"example.com\",\"subdomain\":\"example\",\"tld\":\"com\",\"punycode\":\"example.com\",\"root_domain\":\"example.com\",\"sld\":\"example\",\"valid\":true}],\"reply_to\":[{\"email\":{\"email\":\"user@example.com\",\"local_part\":\"user\",\"domain\":{\"domain\":\"example.com\",\"root_domain\":\"example.com\",\"sld\":\"example\",\"tld\":\"com\",\"valid\":true}}},{\"display_name\":\"Example Display Name\",\"email\":{\"domain\":{\"punycode\":\"example.com\",\"subdomain\":\"sub.example\"}}},{\"display_name\":\"Another Display Name\",\"email\":{\"domain\":{\"punycode\":\"anotherexample.com\",\"subdomain\":\"sub.anotherexample\"}}}]},\"type\":{\"outbound\":true},\"mailbox\":{\"email\":{\"email\":\"user@example.com\",\"local_part\":\"user\",\"domain\":{\"domain\":\"example.com\",\"root_domain\":\"example.com\",\"sld\":\"example\",\"tld\":\"com\",\"valid\":true,\"punycode\":\"xn--example-d4a.com\",\"subdomain\":\"sub\"}}},\"recipients\":{\"to\":[{\"display_name\":\"Alice Johnson\",\"email\":{\"email\":\"user@example.com\",\"local_part\":\"user\",\"domain\":{\"domain\":\"example.com\",\"root_domain\":\"example.com\",\"sld\":\"example\",\"tld\":\"com\",\"valid\":true,\"punycode\":\"xn--example-d4a.net\",\"subdomain\":\"sub\"}}}],\"bcc\":[{\"display_name\":\"John Doe\",\"email\":{\"domain\":{\"domain\":\"example.com\",\"punycode\":\"xn--example-d4a.com\",\"root_domain\":\"example\",\"sld\":\"example\",\"subdomain\":\"sub\",\"tld\":\"com\",\"valid\":true},\"email\":\"john.doe@example.com\",\"local_part\":\"john.doe\"}}],\"cc\":[{\"display_name\":\"Jane Smith\",\"email\":{\"domain\":{\"domain\":\"example.org\",\"punycode\":\"xn--example-d4a.org\",\"root_domain\":\"example\",\"sld\":\"example\",\"subdomain\":\"sub\",\"tld\":\"org\",\"valid\":true},\"email\":\"jane.smith@example.org\",\"local_part\":\"jane.smith\"}}]},\"sender\":{\"display_name\":\"Sublime Security Test\",\"email\":{\"email\":\"testing@sublimesecurity.com\",\"local_part\":\"testing\",\"domain\":{\"domain\":\"sublimesecurity.com\",\"root_domain\":\"sublimesecurity.com\",\"sld\":\"sublimesecurity\",\"tld\":\"com\",\"valid\":true,\"punycode\":\"xn--example-d4a.com\",\"subdomain\":\"sub\"}}},\"subject\":{\"subject\":\"Sublime-Security-Standard-Test-String\"},\"_meta\":{\"id\":\"01911208-633c-7f03-b303-e594d92cf818\",\"canonical_id\":\"2fe271830bbad5fe3a70abbe7a8c0bfe79eb208a76cde267930d19f0e8cea81c\",\"created_at\":\"2024-08-02T07:40:25.135939305Z\",\"effective_at\":\"2024-08-02T07:40:25.135939305Z\"},\"_errors\":[{\"field\":\"Mime-Version\",\"message\":\"No Mime-Version defined in headers\",\"type\":\"missing_header_field\"}]}", + "type": [ + "info" + ] + }, + "input": { + "type": "aws-s3" + }, + "log": { + "file": { + "path": "https://elastic-package-sublime-security-bucket-33881.s3.us-east-1.amazonaws.com/email-message.log" + }, + "offset": 0 + }, + "observer": { + "product": "Sublime Security", + "vendor": "Sublime Security" + }, + "related": { + "hash": [ + "1a2b3c", + "7h8i9j", + "1x2y3z", + "4d5e6f", + "1k2l3m", + "4a5b6c", + "7g8h9i", + "4n5o6p", + "7d8e9f", + "abcdefg" + ], + "hosts": [ + "example.com", + "mail.example.com", + "test.com", + "example", + "example.org", + "sublimesecurity.com" + ], + "ip": [ + "1.128.0.0" + ], + "user": [ + "test", + "user@example.com", + "john.doe@example.com", + "jane.smith@example.org", + "testing@sublimesecurity.com" + ] + }, + "source": { + "domain": "sublimesecurity.com", + "ip": "1.128.0.0", + "subdomain": "sub", + "top_level_domain": "com" + }, + "sublime_security": { + "email_message": { + "attachments": [ + { + "content": { + "id": "abc123", + "transfer_encoding": "base64" + }, + "file": { + "type": "document" + }, + "raw": "JVBERi0xLjMKJcfs4AAQSkZjRgABAQE" + }, + { + "content": { + "id": "xyz456", + "transfer_encoding": "7bit" + }, + "file": { + "type": "image" + }, + "raw": "/9j/4AAQSkZJRgABAQEJVBERi0xLjMKJd" + }, + { + "content": { + "id": "efg789", + "transfer_encoding": "quoted-printable" + }, + "file": { + "type": "text" + }, + "raw": "SGVsbG8gdVsbG8gd29yb29ybGQhVsbG8gd29yb" + } + ], + "body": { + "current_thread": { + "text": "Sublime Security test message." + }, + "html": { + "charset": "utf-8", + "content_transfer_encoding": "base64", + "display_text": "Sublime Security test message.", + "inner_text": "

Sublime Security test message.

", + "raw": "

Sublime Security test message.

" + }, + "ips": [ + { + "ip": "1.128.0.0" + } + ], + "links": [ + { + "display_text": "Click here!", + "display_url": { + "domain": { + "domain": "example.com", + "punycode": "demo", + "root_domain": "example.com", + "sld": "example", + "subdomain": "example", + "tld": "com", + "valid": true + }, + "fragment": "search", + "password": "pass123", + "path": "/test", + "port": 80, + "query_params": "q=elasticsearch", + "rewrite": { + "encoders": [ + "base64" + ], + "original": "demo" + }, + "scheme": "https", + "url": "https://example.com/test?q=elasticsearch#search", + "username": "test" + }, + "mismatched": true + }, + { + "href_url": { + "domain": { + "punycode": "demo", + "root_domain": "example.com", + "sld": "example", + "valid": true + }, + "rewrite": { + "encoders": [ + "base64" + ], + "original": "demo" + } + } + } + ], + "plain": { + "charset": "utf-8", + "content_transfer_encoding": "base64", + "raw": "Sublime Security test message.\n" + } + }, + "errors": [ + { + "field": "Mime-Version", + "message": "No Mime-Version defined in headers", + "type": "missing_header_field" + } + ], + "external": { + "message_id": "2fe271830bbad5fe3a70abbe7a8c0bfe7refe3ffe", + "route_type": "sent", + "spam": false, + "spam_folder": true, + "thread_id": "sample_data" + }, + "headers": { + "auth_summary": { + "dmarc": { + "details": { + "action": "quarantine", + "disposition": "quarantine", + "from": { + "domain": "example.com", + "punycode": "example.com", + "root_domain": "example.com", + "sld": "example", + "subdomain": "example", + "tld": "com", + "valid": true + }, + "policy": "reject", + "sub_policy": "none", + "verdict": "pass", + "version": "1.0" + }, + "pass": true, + "received_hop": 1 + }, + "spf": { + "details": { + "client_ip": { + "ip": "1.128.0.0" + }, + "description": "SPF record found", + "designator": "pass", + "helo": { + "domain": "example.com", + "punycode": "example.com", + "root_domain": "example.com", + "sld": "example", + "subdomain": "example", + "tld": "com", + "valid": true + }, + "server": { + "domain": "mail.example.com", + "punycode": "mail.example.com", + "root_domain": "example.com", + "sld": "example", + "subdomain": "mail", + "tld": "com", + "valid": true + }, + "verdict": "pass" + }, + "error": true, + "pass": true, + "received_hop": 2 + } + }, + "date": "2019-10-21T18:23:24.000Z", + "date_original_offset": "-4", + "delivered_to": { + "domain": { + "punycode": "example.com", + "root_domain": "example.com", + "sld": "example", + "valid": true + }, + "email": "testing@sublimesecurity.com", + "local_part": "testing" + }, + "domains": [ + { + "domain": "test.com", + "punycode": "test.com", + "root_domain": "test.com", + "sld": "test", + "subdomain": "test", + "tld": "com", + "valid": true + }, + { + "domain": "example.com", + "punycode": "example.com", + "root_domain": "example.com", + "sld": "example", + "subdomain": "example", + "tld": "com", + "valid": true + } + ], + "hops": [ + { + "authentication_results": { + "compauth": { + "reason": "reason_value", + "verdict": "pass" + }, + "dkim": "pass", + "dkim_details": { + "algorithm": "rsa-sha256", + "body_hash": "abcdefg", + "domain": "example.com", + "headers": "from, to, subject", + "instance": "example.com", + "selector": "abcdefg", + "signature": "abcdefg", + "type": "dkim", + "version": "1.0" + }, + "dmarc": "pass", + "dmarc_details": { + "action": "quarantine", + "disposition": "quarantine", + "from": { + "domain": "example.com", + "punycode": "example.com", + "root_domain": "example.com", + "sld": "example", + "subdomain": "example", + "tld": "com", + "valid": true + }, + "policy": "reject", + "sub_policy": "none", + "verdict": "pass", + "version": "1.0" + }, + "instance": "example.com", + "server": { + "domain": "mail.example.com", + "punycode": "mail.example.com", + "root_domain": "example.com", + "sld": "example", + "subdomain": "mail", + "tld": "com", + "valid": true + }, + "spf": "pass", + "spf_details": { + "client_ip": { + "ip": "1.128.0.0" + }, + "description": "SPF record found", + "designator": "pass", + "helo": { + "domain": "example.com", + "punycode": "example.com", + "root_domain": "example.com", + "sld": "example", + "subdomain": "example", + "tld": "com", + "valid": true + }, + "server": { + "domain": "mail.example.com", + "root_domain": "example.com", + "sld": "example", + "subdomain": "mail", + "tld": "com", + "valid": true + }, + "verdict": "pass" + }, + "type": "spf" + }, + "fields": [ + { + "name": "To", + "position": 0, + "to": "user@example.com", + "value": "user@example.com" + }, + { + "name": "Subject", + "position": 1, + "subject": "Sublime-Security-Standard-Test-String", + "value": "Sublime-Security-Standard-Test-String" + }, + { + "date": "Mon, 21 Oct 2019 14:23:24 -0400", + "name": "Date", + "position": 2, + "value": "Mon, 21 Oct 2019 14:23:24 -0400" + }, + { + "from": "Sublime Security Test ", + "name": "From", + "position": 3, + "value": "Sublime Security Test " + } + ], + "index": 0, + "received": { + "additional": { + "raw": "Authentication successful" + }, + "id": { + "raw": "msg-12345" + }, + "link": { + "raw": "https://mail.example.com/message/12345" + }, + "mailbox": { + "raw": "user@example.com" + }, + "protocol": { + "raw": "IMAP" + }, + "server": { + "raw": "imap.example.com" + }, + "source": { + "raw": "81.2.69.144" + }, + "time": "2019-10-21T18:23:24.000Z", + "zone_offset": "+00:00" + }, + "received_spf": { + "client_ip": { + "ip": "1.128.0.0" + }, + "description": "SPF record found", + "designator": "pass", + "helo": { + "domain": "example.com", + "punycode": "example.com", + "root_domain": "example.com", + "sld": "example", + "subdomain": "example", + "tld": "com", + "valid": true + }, + "server": { + "domain": "mail.example.com", + "punycode": "mail.example.com", + "root_domain": "example.com", + "sld": "example", + "subdomain": "mail", + "tld": "com", + "valid": true + }, + "verdict": "pass" + }, + "signature": { + "algorithm": "rsa-sha256", + "body_hash": "b9c4a3f9d93d9a38bdf8c47a8f2d2c79ec1d8b1f", + "domain": "example.com", + "headers": "from:to:subject:date", + "instance": "123456", + "selector": "default", + "signature": "d2abf9d6c8f4b8d68d8f3f7b6f9d3b8e6a8c2b3a9f4b8d7b9d3b6a8f9c3b4e5f", + "type": "spf", + "version": "1" + } + } + ], + "in_reply_to": "in_reply_to_value", + "ips": [ + { + "ip": "1.128.0.0" + } + ], + "references": [ + "test1", + "test2" + ], + "reply_to": [ + { + "email": { + "domain": { + "domain": "example.com", + "root_domain": "example.com", + "sld": "example", + "tld": "com", + "valid": true + }, + "local_part": "user" + } + }, + { + "display_name": "Example Display Name", + "email": { + "domain": { + "punycode": "example.com", + "subdomain": "sub.example" + } + } + }, + { + "display_name": "Another Display Name", + "email": { + "domain": { + "punycode": "anotherexample.com", + "subdomain": "sub.anotherexample" + } + } + } + ], + "return_path": { + "domain": { + "domain": "example.com", + "punycode": "xn--example-d4a.com", + "root_domain": "example", + "sld": "example", + "subdomain": "sub", + "tld": "com", + "valid": true + }, + "email": "user@example.com", + "local_part": "user" + }, + "x_authenticated_domain": { + "domain": "example.com", + "punycode": "xn--example-d4a.com", + "root_domain": "example.com", + "sld": "example", + "subdomain": "sub", + "tld": "com", + "valid": true + }, + "x_authenticated_sender": { + "domain": { + "domain": "example.com", + "punycode": "example.com", + "root_domain": "example.com", + "sld": "example", + "subdomain": "sub", + "tld": "com", + "valid": true + }, + "email": "user@example.com", + "local_part": "user" + }, + "x_originating_ip": { + "ip": "1.128.0.0" + }, + "x_secure_server_account": "account_value", + "x_sender": { + "domain": { + "domain": "example.com", + "punycode": "example.com", + "root_domain": "example.com", + "sld": "example", + "subdomain": "sub", + "tld": "com", + "valid": true + }, + "email": "user@example.com", + "local_part": "user" + } + }, + "mailbox": { + "email": { + "domain": { + "domain": "example.com", + "punycode": "xn--example-d4a.com", + "root_domain": "example.com", + "sld": "example", + "subdomain": "sub", + "tld": "com", + "valid": true + }, + "local_part": "user", + "value": "user@example.com" + } + }, + "meta": { + "canonical_id": "2fe271830bbad5fe3a70abbe7a8c0bfe79eb208a76cde267930d19f0e8cea81c", + "effective_at": "2024-08-02T07:40:25.135Z" + }, + "recipients": { + "bcc": [ + { + "display_name": "John Doe", + "email": { + "domain": { + "domain": "example.com", + "punycode": "xn--example-d4a.com", + "root_domain": "example", + "sld": "example", + "subdomain": "sub", + "tld": "com", + "valid": true + }, + "local_part": "john.doe" + } + } + ], + "cc": [ + { + "display_name": "Jane Smith", + "email": { + "domain": { + "domain": "example.org", + "punycode": "xn--example-d4a.org", + "root_domain": "example", + "sld": "example", + "subdomain": "sub", + "tld": "org", + "valid": true + }, + "local_part": "jane.smith" + } + } + ], + "to": [ + { + "display_name": "Alice Johnson", + "email": { + "domain": { + "domain": "example.com", + "punycode": "xn--example-d4a.net", + "root_domain": "example.com", + "sld": "example", + "subdomain": "sub", + "tld": "com", + "valid": true + }, + "local_part": "user" + } + } + ] + }, + "sender": { + "display_name": "Sublime Security Test", + "email": { + "domain": { + "punycode": "xn--example-d4a.com", + "root_domain": "sublimesecurity.com", + "sld": "sublimesecurity", + "valid": true + }, + "local_part": "testing" + } + }, + "type": { + "outbound": true + } + } + }, + "tags": [ + "collect_sqs_logs", + "preserve_original_event", + "forwarded", + "sublime_security-email_message" + ], + "url": [ + { + "domain": "example.com", + "fragment": "search", + "full": "https://example.com/test?q=elasticsearch#search", + "password": "pass123", + "path": "/test", + "port": 80, + "query": "q=elasticsearch", + "scheme": "https", + "subdomain": "example", + "top_level_domain": "com", + "username": "test" + } + ], + "user_agent": { + "device": { + "name": "Other" + }, + "name": "Other", + "original": "MyCustomMailer" + } +} +``` + +**Exported fields** + +| Field | Description | Type | +|---|---|---| +| @timestamp | Event timestamp. | date | +| aws.s3.bucket.arn | The AWS S3 bucket ARN. | keyword | +| aws.s3.bucket.name | The AWS S3 bucket name. | keyword | +| aws.s3.object.key | The AWS S3 Object key. | keyword | +| data_stream.dataset | Data stream dataset. | constant_keyword | +| data_stream.namespace | Data stream namespace. | constant_keyword | +| data_stream.type | Data stream type. | constant_keyword | +| event.dataset | Event dataset. | constant_keyword | +| event.module | Event module. | constant_keyword | +| input.type | Type of filebeat input. | keyword | +| log.offset | Log offset. | long | +| sublime_security.email_message.attachments.content.id | Content-ID extracted from the MIME payload. | keyword | +| sublime_security.email_message.attachments.content.transfer_encoding | Content-Transfer-Encoding extracted from the MIME payload. | keyword | +| sublime_security.email_message.attachments.content.type | Content-Type extracted from the MIME payload. | keyword | +| sublime_security.email_message.attachments.file.extension | File extension from context such as headers. | keyword | +| sublime_security.email_message.attachments.file.name | File name. | keyword | +| sublime_security.email_message.attachments.file.type | File type determined by looking at the magic bytes in the file. | keyword | +| sublime_security.email_message.attachments.md5 | MD5 hash of the raw contents. | keyword | +| sublime_security.email_message.attachments.raw | Base64 encoded source of the file. | keyword | +| sublime_security.email_message.attachments.sha1 | SHA1 hash of the raw contents. | keyword | +| sublime_security.email_message.attachments.sha256 | SHA256 hash of the raw contents. | keyword | +| sublime_security.email_message.attachments.size | Size of the file in bytes. | long | +| sublime_security.email_message.body.current_thread.text | The text content from the latest reply/forward in a message thread. This typically excludes content from forwarded messages and warning banners. | keyword | +| sublime_security.email_message.body.html.charset | charset of the text/[subtype]. | keyword | +| sublime_security.email_message.body.html.content_transfer_encoding | Content-Transfer-Encoding of the text/[subtype]. | keyword | +| sublime_security.email_message.body.html.display_text | Visible text of the HTML document, with invisible characters removed and non-ASCII characters converted to ASCII spaces. | keyword | +| sublime_security.email_message.body.html.inner_text | Inner text of the HTML document that doesn't include HTML tags. | keyword | +| sublime_security.email_message.body.html.raw | Decoded raw content of a body text type (text/[subtype] section). | keyword | +| sublime_security.email_message.body.ips.ip | The raw IP. | ip | +| sublime_security.email_message.body.links.display_text | The text of a hyperlink, if it's not a URL. | keyword | +| sublime_security.email_message.body.links.display_url.domain.domain | The fully qualified domain name (FQDN). This may not always be routable, e.g. when an email address contains a domain that is just a TLD with no SLD, e.g. foo@WIN-bar. | keyword | +| sublime_security.email_message.body.links.display_url.domain.punycode | Interpreted punycode if the domain starts with xn--. For example, if 'domain' is 'xn--ublimesecurity-4xc.com' then 'punycode' is śublimesecurity.com. | keyword | +| sublime_security.email_message.body.links.display_url.domain.root_domain | The root domain, including the TLD. | keyword | +| sublime_security.email_message.body.links.display_url.domain.sld | Second-level domain, e.g. 'windows' for the domain 'windows.net'. | keyword | +| sublime_security.email_message.body.links.display_url.domain.subdomain | Subdomain, e.g. 'drive' for the domain 'drive.google.com'. | keyword | +| sublime_security.email_message.body.links.display_url.domain.tld | The domain's top-level domain. E.g. the TLD of google.com is 'com'. | keyword | +| sublime_security.email_message.body.links.display_url.domain.valid | Whether the domain is valid. | boolean | +| sublime_security.email_message.body.links.display_url.fragment | Fragment identifier; the text following the # in the href_url (also called the anchor tag). | keyword | +| sublime_security.email_message.body.links.display_url.password | The password specified before the domain name. | keyword | +| sublime_security.email_message.body.links.display_url.path | Everything after the TLD and before the query parameters. | keyword | +| sublime_security.email_message.body.links.display_url.port | The port used for the href_url. If no explicit port is set, the port will be inferred from the protocol. | long | +| sublime_security.email_message.body.links.display_url.query_params | The query parameters of the href_url. | keyword | +| sublime_security.email_message.body.links.display_url.rewrite.encoders | List of detected URL rewrite encoders while unraveling the URL. | keyword | +| sublime_security.email_message.body.links.display_url.rewrite.original | Original URL without any unraveling URL rewrites. | keyword | +| sublime_security.email_message.body.links.display_url.scheme | Protocol for the href_url request, e.g. http. | keyword | +| sublime_security.email_message.body.links.display_url.url | Full URL. | keyword | +| sublime_security.email_message.body.links.display_url.username | The username specified before the domain name of the href_url. | keyword | +| sublime_security.email_message.body.links.href_url.domain.domain | The fully qualified domain name (FQDN). This may not always be routable, e.g. when an email address contains a domain that is just a TLD with no SLD, e.g. foo@WIN-bar. | keyword | +| sublime_security.email_message.body.links.href_url.domain.punycode | Interpreted punycode if the domain starts with xn--. For example, if 'domain' is 'xn--ublimesecurity-4xc.com' then 'punycode' is śublimesecurity.com. | keyword | +| sublime_security.email_message.body.links.href_url.domain.root_domain | The root domain, including the TLD. | keyword | +| sublime_security.email_message.body.links.href_url.domain.sld | Second-level domain, e.g. 'windows' for the domain 'windows.net'. | keyword | +| sublime_security.email_message.body.links.href_url.domain.subdomain | Subdomain, e.g. 'drive' for the domain 'drive.google.com'. | keyword | +| sublime_security.email_message.body.links.href_url.domain.tld | The domain's top-level domain. E.g. the TLD of google.com is 'com'. | keyword | +| sublime_security.email_message.body.links.href_url.domain.valid | Whether the domain is valid. | boolean | +| sublime_security.email_message.body.links.href_url.fragment | Fragment identifier; the text following the # in the href_url (also called the anchor tag). | keyword | +| sublime_security.email_message.body.links.href_url.password | The password specified before the domain name. | keyword | +| sublime_security.email_message.body.links.href_url.path | Everything after the TLD and before the query parameters. | keyword | +| sublime_security.email_message.body.links.href_url.port | The port used for the href_url. If no explicit port is set, the port will be inferred from the protocol. | long | +| sublime_security.email_message.body.links.href_url.query_params | The query parameters of the href_url. | keyword | +| sublime_security.email_message.body.links.href_url.rewrite.encoders | List of detected URL rewrite encoders while unraveling the URL. | keyword | +| sublime_security.email_message.body.links.href_url.rewrite.original | Original URL without any unraveling URL rewrites. | keyword | +| sublime_security.email_message.body.links.href_url.scheme | Protocol for the href_url request, e.g. http. | keyword | +| sublime_security.email_message.body.links.href_url.url | Full URL. | keyword | +| sublime_security.email_message.body.links.href_url.username | The username specified before the domain name of the href_url. | keyword | +| sublime_security.email_message.body.links.mismatched | Whether the display URL and href URL root domains are mismatched (i.e. .href_url.domain.root_domain != .display_url.domain.root_domain, where both are not null and valid domains). | boolean | +| sublime_security.email_message.body.plain.charset | charset of the text/[subtype]. | keyword | +| sublime_security.email_message.body.plain.content_transfer_encoding | Content-Transfer-Encoding of the text/[subtype]. | keyword | +| sublime_security.email_message.body.plain.raw | Decoded raw content of a body text type (text/[subtype] section). | keyword | +| sublime_security.email_message.errors | Non-fatal errors while parsing MDM. | object | +| sublime_security.email_message.external.created_at | The created time of the message as provided by the cloud API (G Suite or Office 365) or other external source. This is typically the time the external source received the message. | date | +| sublime_security.email_message.external.message_id | The message ID as provided by the cloud API (G Suite or Office 365) or other external source. | keyword | +| sublime_security.email_message.external.route_type | whether the message was sent or received. | keyword | +| sublime_security.email_message.external.spam | The upstream mail gateway determined the message to be spam. For cloud API providers, this will be the same as spam_folder. For other implementation methods like transport rules, this will be determined by message header values (e.g. X-SPAM) if supported. | boolean | +| sublime_security.email_message.external.spam_folder | The message arrived in the user's spam folder. This only applies to cloud APIs (G Suite or Office 365). | boolean | +| sublime_security.email_message.external.thread_id | The thread/conversation's unique ID as provided by the cloud API (G Suite or Office 365). | keyword | +| sublime_security.email_message.headers.auth_summary.dmarc.details.action | Indicates the action taken by the spam filter based on the results of the DMARC check. For more information see https://docs.microsoft.com/en-us/microsoft-365/security/office-365-security/anti-spam-message-headers?view=o365-worldwide#authentication-results-message-header-fields. | keyword | +| sublime_security.email_message.headers.auth_summary.dmarc.details.disposition | Gmail-applied policy. | keyword | +| sublime_security.email_message.headers.auth_summary.dmarc.details.from.domain | The fully qualified domain name (FQDN). This may not always be routable, e.g. when an email address contains a domain that is just a TLD with no SLD, e.g. foo@WIN-bar. | keyword | +| sublime_security.email_message.headers.auth_summary.dmarc.details.from.punycode | Interpreted punycode if the domain starts with xn--. For example, if 'domain' is 'xn--ublimesecurity-4xc.com' then 'punycode' is śublimesecurity.com. | keyword | +| sublime_security.email_message.headers.auth_summary.dmarc.details.from.root_domain | The root domain, including the TLD. | keyword | +| sublime_security.email_message.headers.auth_summary.dmarc.details.from.sld | Second-level domain, e.g. 'windows' for the domain 'windows.net'. | keyword | +| sublime_security.email_message.headers.auth_summary.dmarc.details.from.subdomain | Subdomain, e.g. 'drive' for the domain 'drive.google.com'. | keyword | +| sublime_security.email_message.headers.auth_summary.dmarc.details.from.tld | The domain's top-level domain. E.g. the TLD of google.com is 'com'. | keyword | +| sublime_security.email_message.headers.auth_summary.dmarc.details.from.valid | Whether the domain is valid. | boolean | +| sublime_security.email_message.headers.auth_summary.dmarc.details.policy | Policy for the organizational domain. | keyword | +| sublime_security.email_message.headers.auth_summary.dmarc.details.sub_policy | Policy for the subdomain of the organizational domain. | keyword | +| sublime_security.email_message.headers.auth_summary.dmarc.details.verdict | Describes the results of the DMARC check for the message. | keyword | +| sublime_security.email_message.headers.auth_summary.dmarc.details.version | DMARC version. | keyword | +| sublime_security.email_message.headers.auth_summary.dmarc.pass | Whether the DMARC check passed. | boolean | +| sublime_security.email_message.headers.auth_summary.dmarc.received_hop | The lowest hop at which the DMARC check was made. | long | +| sublime_security.email_message.headers.auth_summary.spf.details.client_ip.ip | The raw IP. | ip | +| sublime_security.email_message.headers.auth_summary.spf.details.description | Verbose description of the SPF verdict. | keyword | +| sublime_security.email_message.headers.auth_summary.spf.details.designator | Email or domain of the designating body. | keyword | +| sublime_security.email_message.headers.auth_summary.spf.details.helo.domain | The fully qualified domain name (FQDN). This may not always be routable, e.g. when an email address contains a domain that is just a TLD with no SLD, e.g. foo@WIN-bar. | keyword | +| sublime_security.email_message.headers.auth_summary.spf.details.helo.punycode | Interpreted punycode if the domain starts with xn--. For example, if 'domain' is 'xn--ublimesecurity-4xc.com' then 'punycode' is śublimesecurity.com. | keyword | +| sublime_security.email_message.headers.auth_summary.spf.details.helo.root_domain | The root domain, including the TLD. | keyword | +| sublime_security.email_message.headers.auth_summary.spf.details.helo.sld | Second-level domain, e.g. 'windows' for the domain 'windows.net'. | keyword | +| sublime_security.email_message.headers.auth_summary.spf.details.helo.subdomain | Subdomain, e.g. 'drive' for the domain 'drive.google.com'. | keyword | +| sublime_security.email_message.headers.auth_summary.spf.details.helo.tld | The domain's top-level domain. E.g. the TLD of google.com is 'com'. | keyword | +| sublime_security.email_message.headers.auth_summary.spf.details.helo.valid | Whether the domain is valid. | boolean | +| sublime_security.email_message.headers.auth_summary.spf.details.server.domain | The fully qualified domain name (FQDN). This may not always be routable, e.g. when an email address contains a domain that is just a TLD with no SLD, e.g. foo@WIN-bar. | keyword | +| sublime_security.email_message.headers.auth_summary.spf.details.server.punycode | Interpreted punycode if the domain starts with xn--. For example, if 'domain' is 'xn--ublimesecurity-4xc.com' then 'punycode' is śublimesecurity.com. | keyword | +| sublime_security.email_message.headers.auth_summary.spf.details.server.root_domain | The root domain, including the TLD. | keyword | +| sublime_security.email_message.headers.auth_summary.spf.details.server.sld | Second-level domain, e.g. 'windows' for the domain 'windows.net'. | keyword | +| sublime_security.email_message.headers.auth_summary.spf.details.server.subdomain | Subdomain, e.g. 'drive' for the domain 'drive.google.com'. | keyword | +| sublime_security.email_message.headers.auth_summary.spf.details.server.tld | The domain's top-level domain. E.g. the TLD of google.com is 'com'. | keyword | +| sublime_security.email_message.headers.auth_summary.spf.details.server.valid | Whether the domain is valid. | boolean | +| sublime_security.email_message.headers.auth_summary.spf.details.verdict | Verdict of the SPF. | keyword | +| sublime_security.email_message.headers.auth_summary.spf.error | Whether the SPF check errored. | boolean | +| sublime_security.email_message.headers.auth_summary.spf.pass | Whether the SPF check passed. | boolean | +| sublime_security.email_message.headers.auth_summary.spf.received_hop | The lowest hop at which the SPF check was made. | long | +| sublime_security.email_message.headers.date | Date the email was sent in UTC. | date | +| sublime_security.email_message.headers.date_original_offset | UTC timezone offset of the sender. | keyword | +| sublime_security.email_message.headers.delivered_to.domain.domain | The fully qualified domain name (FQDN). This may not always be routable, e.g. when an email address contains a domain that is just a TLD with no SLD, e.g. foo@WIN-bar. | keyword | +| sublime_security.email_message.headers.delivered_to.domain.punycode | Interpreted punycode if the domain starts with xn--. For example, if 'domain' is 'xn--ublimesecurity-4xc.com' then 'punycode' is śublimesecurity.com. | keyword | +| sublime_security.email_message.headers.delivered_to.domain.root_domain | The root domain, including the TLD. | keyword | +| sublime_security.email_message.headers.delivered_to.domain.sld | Second-level domain, e.g. 'windows' for the domain 'windows.net'. | keyword | +| sublime_security.email_message.headers.delivered_to.domain.subdomain | Subdomain, e.g. 'drive' for the domain 'drive.google.com'. | keyword | +| sublime_security.email_message.headers.delivered_to.domain.tld | The domain's top-level domain. E.g. the TLD of google.com is 'com'. | keyword | +| sublime_security.email_message.headers.delivered_to.domain.valid | Whether the domain is valid. | boolean | +| sublime_security.email_message.headers.delivered_to.email | Full email address. | keyword | +| sublime_security.email_message.headers.delivered_to.local_part | Local-part, i.e. before the @. | keyword | +| sublime_security.email_message.headers.domains.domain | The fully qualified domain name (FQDN). This may not always be routable, e.g. when an email address contains a domain that is just a TLD with no SLD, e.g. foo@WIN-bar. | keyword | +| sublime_security.email_message.headers.domains.punycode | Interpreted punycode if the domain starts with xn--. For example, if 'domain' is 'xn--ublimesecurity-4xc.com' then 'punycode' is śublimesecurity.com. | keyword | +| sublime_security.email_message.headers.domains.root_domain | The root domain, including the TLD. | keyword | +| sublime_security.email_message.headers.domains.sld | Second-level domain, e.g. 'windows' for the domain 'windows.net'. | keyword | +| sublime_security.email_message.headers.domains.subdomain | Subdomain, e.g. 'drive' for the domain 'drive.google.com'. | keyword | +| sublime_security.email_message.headers.domains.tld | The domain's top-level domain. E.g. the TLD of google.com is 'com'. | keyword | +| sublime_security.email_message.headers.domains.valid | Whether the domain is valid. | boolean | +| sublime_security.email_message.headers.hops.authentication_results.compauth.reason | Reason for the verdict. | keyword | +| sublime_security.email_message.headers.hops.authentication_results.compauth.verdict | Verdict of the compauth. | keyword | +| sublime_security.email_message.headers.hops.authentication_results.dkim | Verdict of the Domain Keys Identified Mail check. | keyword | +| sublime_security.email_message.headers.hops.authentication_results.dkim_details.algorithm | Signing algorithm. | keyword | +| sublime_security.email_message.headers.hops.authentication_results.dkim_details.body_hash | Body Hash. | keyword | +| sublime_security.email_message.headers.hops.authentication_results.dkim_details.domain | Domain identified in the DKIM signature if any. This is the domain that's queried for the public key. | keyword | +| sublime_security.email_message.headers.hops.authentication_results.dkim_details.headers | Header fields signed by the algorithm. | keyword | +| sublime_security.email_message.headers.hops.authentication_results.dkim_details.instance | Instance number of this signature (if ARC). | keyword | +| sublime_security.email_message.headers.hops.authentication_results.dkim_details.selector | Selector. | keyword | +| sublime_security.email_message.headers.hops.authentication_results.dkim_details.signature | Signature of headers and body. | keyword | +| sublime_security.email_message.headers.hops.authentication_results.dkim_details.type | The type of signature, derived from the field name. | keyword | +| sublime_security.email_message.headers.hops.authentication_results.dkim_details.version | Version. | keyword | +| sublime_security.email_message.headers.hops.authentication_results.dmarc | Verdict of the Domain-based Message Authentication, Reporting & Conformance check. | keyword | +| sublime_security.email_message.headers.hops.authentication_results.dmarc_details.action | Indicates the action taken by the spam filter based on the results of the DMARC check. For more information see https://docs.microsoft.com/en-us/microsoft-365/security/office-365-security/anti-spam-message-headers?view=o365-worldwide#authentication-results-message-header-fields. | keyword | +| sublime_security.email_message.headers.hops.authentication_results.dmarc_details.disposition | Gmail-applied policy. | keyword | +| sublime_security.email_message.headers.hops.authentication_results.dmarc_details.from.domain | The fully qualified domain name (FQDN). This may not always be routable, e.g. when an email address contains a domain that is just a TLD with no SLD, e.g. foo@WIN-bar. | keyword | +| sublime_security.email_message.headers.hops.authentication_results.dmarc_details.from.punycode | Interpreted punycode if the domain starts with xn--. For example, if 'domain' is 'xn--ublimesecurity-4xc.com' then 'punycode' is śublimesecurity.com. | keyword | +| sublime_security.email_message.headers.hops.authentication_results.dmarc_details.from.root_domain | The root domain, including the TLD. | keyword | +| sublime_security.email_message.headers.hops.authentication_results.dmarc_details.from.sld | Second-level domain, e.g. 'windows' for the domain 'windows.net'. | keyword | +| sublime_security.email_message.headers.hops.authentication_results.dmarc_details.from.subdomain | Subdomain, e.g. 'drive' for the domain 'drive.google.com'. | keyword | +| sublime_security.email_message.headers.hops.authentication_results.dmarc_details.from.tld | The domain's top-level domain. E.g. the TLD of google.com is 'com'. | keyword | +| sublime_security.email_message.headers.hops.authentication_results.dmarc_details.from.valid | Whether the domain is valid. | boolean | +| sublime_security.email_message.headers.hops.authentication_results.dmarc_details.policy | Policy for the organizational domain. | keyword | +| sublime_security.email_message.headers.hops.authentication_results.dmarc_details.sub_policy | Policy for the subdomain of the organizational domain. | keyword | +| sublime_security.email_message.headers.hops.authentication_results.dmarc_details.verdict | Describes the results of the DMARC check for the message. | keyword | +| sublime_security.email_message.headers.hops.authentication_results.dmarc_details.version | DMARC version. | keyword | +| sublime_security.email_message.headers.hops.authentication_results.instance | Instance number of this auth result (if ARC). | keyword | +| sublime_security.email_message.headers.hops.authentication_results.server.domain | The fully qualified domain name (FQDN). This may not always be routable, e.g. when an email address contains a domain that is just a TLD with no SLD, e.g. foo@WIN-bar. | keyword | +| sublime_security.email_message.headers.hops.authentication_results.server.punycode | Interpreted punycode if the domain starts with xn--. For example, if 'domain' is 'xn--ublimesecurity-4xc.com' then 'punycode' is śublimesecurity.com. | keyword | +| sublime_security.email_message.headers.hops.authentication_results.server.root_domain | The root domain, including the TLD. | keyword | +| sublime_security.email_message.headers.hops.authentication_results.server.sld | Second-level domain, e.g. 'windows' for the domain 'windows.net'. | keyword | +| sublime_security.email_message.headers.hops.authentication_results.server.subdomain | Subdomain, e.g. 'drive' for the domain 'drive.google.com'. | keyword | +| sublime_security.email_message.headers.hops.authentication_results.server.tld | The domain's top-level domain. E.g. the TLD of google.com is 'com'. | keyword | +| sublime_security.email_message.headers.hops.authentication_results.server.valid | Whether the domain is valid. | boolean | +| sublime_security.email_message.headers.hops.authentication_results.spf | Verdict of the Sender Policy Framework. | keyword | +| sublime_security.email_message.headers.hops.authentication_results.spf_details.client_ip.ip | The raw IP. | ip | +| sublime_security.email_message.headers.hops.authentication_results.spf_details.description | Verbose description of the SPF verdict. | keyword | +| sublime_security.email_message.headers.hops.authentication_results.spf_details.designator | Email or domain of the designating body. | keyword | +| sublime_security.email_message.headers.hops.authentication_results.spf_details.helo.domain | The fully qualified domain name (FQDN). This may not always be routable, e.g. when an email address contains a domain that is just a TLD with no SLD, e.g. foo@WIN-bar. | keyword | +| sublime_security.email_message.headers.hops.authentication_results.spf_details.helo.punycode | Interpreted punycode if the domain starts with xn--. For example, if 'domain' is 'xn--ublimesecurity-4xc.com' then 'punycode' is śublimesecurity.com. | keyword | +| sublime_security.email_message.headers.hops.authentication_results.spf_details.helo.root_domain | The root domain, including the TLD. | keyword | +| sublime_security.email_message.headers.hops.authentication_results.spf_details.helo.sld | Second-level domain, e.g. 'windows' for the domain 'windows.net'. | keyword | +| sublime_security.email_message.headers.hops.authentication_results.spf_details.helo.subdomain | Subdomain, e.g. 'drive' for the domain 'drive.google.com'. | keyword | +| sublime_security.email_message.headers.hops.authentication_results.spf_details.helo.tld | The domain's top-level domain. E.g. the TLD of google.com is 'com'. | keyword | +| sublime_security.email_message.headers.hops.authentication_results.spf_details.helo.valid | Whether the domain is valid. | boolean | +| sublime_security.email_message.headers.hops.authentication_results.spf_details.server.domain | The fully qualified domain name (FQDN). This may not always be routable, e.g. when an email address contains a domain that is just a TLD with no SLD, e.g. foo@WIN-bar. | keyword | +| sublime_security.email_message.headers.hops.authentication_results.spf_details.server.punycode | Interpreted punycode if the domain starts with xn--. For example, if 'domain' is 'xn--ublimesecurity-4xc.com' then 'punycode' is śublimesecurity.com. | keyword | +| sublime_security.email_message.headers.hops.authentication_results.spf_details.server.root_domain | The root domain, including the TLD. | keyword | +| sublime_security.email_message.headers.hops.authentication_results.spf_details.server.sld | Second-level domain, e.g. 'windows' for the domain 'windows.net'. | keyword | +| sublime_security.email_message.headers.hops.authentication_results.spf_details.server.subdomain | Subdomain, e.g. 'drive' for the domain 'drive.google.com'. | keyword | +| sublime_security.email_message.headers.hops.authentication_results.spf_details.server.tld | The domain's top-level domain. E.g. the TLD of google.com is 'com'. | keyword | +| sublime_security.email_message.headers.hops.authentication_results.spf_details.server.valid | Whether the domain is valid. | boolean | +| sublime_security.email_message.headers.hops.authentication_results.spf_details.verdict | Verdict of the SPF. | keyword | +| sublime_security.email_message.headers.hops.authentication_results.type | The type of authentication result, derived from the field name. | keyword | +| sublime_security.email_message.headers.hops.fields | | object | +| sublime_security.email_message.headers.hops.index | Index indicates the order in which a hop occurred from sender to recipient. | long | +| sublime_security.email_message.headers.hops.received.additional.raw | The raw string for remaining additional clauses, such as transport information. | keyword | +| sublime_security.email_message.headers.hops.received.id.raw | The raw string of 'id' section. | keyword | +| sublime_security.email_message.headers.hops.received.link.raw | The raw string of 'via' section. | keyword | +| sublime_security.email_message.headers.hops.received.mailbox.raw | The raw string of 'for' section. | keyword | +| sublime_security.email_message.headers.hops.received.protocol.raw | The raw string of 'with' section. | keyword | +| sublime_security.email_message.headers.hops.received.server.raw | The raw string of 'by' section. | keyword | +| sublime_security.email_message.headers.hops.received.source.raw | The raw string of 'from' section. | keyword | +| sublime_security.email_message.headers.hops.received.time | Time parsed from the Received header. | date | +| sublime_security.email_message.headers.hops.received.zone_offset | Timezone offset parsed from the Received header. | keyword | +| sublime_security.email_message.headers.hops.received_spf.client_ip.ip | The raw IP. | ip | +| sublime_security.email_message.headers.hops.received_spf.description | Verbose description of the SPF verdict. | keyword | +| sublime_security.email_message.headers.hops.received_spf.designator | Email or domain of the designating body. | keyword | +| sublime_security.email_message.headers.hops.received_spf.helo.domain | The fully qualified domain name (FQDN). This may not always be routable, e.g. when an email address contains a domain that is just a TLD with no SLD, e.g. foo@WIN-bar. | keyword | +| sublime_security.email_message.headers.hops.received_spf.helo.punycode | Interpreted punycode if the domain starts with xn--. For example, if 'domain' is 'xn--ublimesecurity-4xc.com' then 'punycode' is śublimesecurity.com. | keyword | +| sublime_security.email_message.headers.hops.received_spf.helo.root_domain | The root domain, including the TLD. | keyword | +| sublime_security.email_message.headers.hops.received_spf.helo.sld | Second-level domain, e.g. 'windows' for the domain 'windows.net'. | keyword | +| sublime_security.email_message.headers.hops.received_spf.helo.subdomain | Subdomain, e.g. 'drive' for the domain 'drive.google.com'. | keyword | +| sublime_security.email_message.headers.hops.received_spf.helo.tld | The domain's top-level domain. E.g. the TLD of google.com is 'com'. | keyword | +| sublime_security.email_message.headers.hops.received_spf.helo.valid | Whether the domain is valid. | boolean | +| sublime_security.email_message.headers.hops.received_spf.server.domain | The fully qualified domain name (FQDN). This may not always be routable, e.g. when an email address contains a domain that is just a TLD with no SLD, e.g. foo@WIN-bar. | keyword | +| sublime_security.email_message.headers.hops.received_spf.server.punycode | Interpreted punycode if the domain starts with xn--. For example, if 'domain' is 'xn--ublimesecurity-4xc.com' then 'punycode' is śublimesecurity.com. | keyword | +| sublime_security.email_message.headers.hops.received_spf.server.root_domain | The root domain, including the TLD. | keyword | +| sublime_security.email_message.headers.hops.received_spf.server.sld | Second-level domain, e.g. 'windows' for the domain 'windows.net'. | keyword | +| sublime_security.email_message.headers.hops.received_spf.server.subdomain | Subdomain, e.g. 'drive' for the domain 'drive.google.com'. | keyword | +| sublime_security.email_message.headers.hops.received_spf.server.tld | The domain's top-level domain. E.g. the TLD of google.com is 'com'. | keyword | +| sublime_security.email_message.headers.hops.received_spf.server.valid | Whether the domain is valid. | boolean | +| sublime_security.email_message.headers.hops.received_spf.verdict | Verdict of the SPF. | keyword | +| sublime_security.email_message.headers.hops.signature.algorithm | Signing algorithm. | keyword | +| sublime_security.email_message.headers.hops.signature.body_hash | Body Hash. | keyword | +| sublime_security.email_message.headers.hops.signature.domain | Domain identified in the DKIM signature if any. This is the domain that's queried for the public key. | keyword | +| sublime_security.email_message.headers.hops.signature.headers | Header fields signed by the algorithm. | keyword | +| sublime_security.email_message.headers.hops.signature.instance | Instance number of this signature (if ARC). | keyword | +| sublime_security.email_message.headers.hops.signature.selector | Selector. | keyword | +| sublime_security.email_message.headers.hops.signature.signature | Signature of headers and body. | keyword | +| sublime_security.email_message.headers.hops.signature.type | The type of signature, derived from the field name. | keyword | +| sublime_security.email_message.headers.hops.signature.version | Version. | keyword | +| sublime_security.email_message.headers.in_reply_to | In-Reply-To header value which identifies its parent message if exists. | keyword | +| sublime_security.email_message.headers.ips.ip | The raw IP. | keyword | +| sublime_security.email_message.headers.mailer | X-Mailer or User-Agent extracted from headers. | keyword | +| sublime_security.email_message.headers.message_id | Message-ID extracted from the header. | keyword | +| sublime_security.email_message.headers.references | The Message-IDs of the other messages within this chain. | keyword | +| sublime_security.email_message.headers.reply_to.display_name | Display name. | keyword | +| sublime_security.email_message.headers.reply_to.email.domain.domain | The fully qualified domain name (FQDN). This may not always be routable, e.g. when an email address contains a domain that is just a TLD with no SLD, e.g. foo@WIN-bar. | keyword | +| sublime_security.email_message.headers.reply_to.email.domain.punycode | Interpreted punycode if the domain starts with xn--. For example, if 'domain' is 'xn--ublimesecurity-4xc.com' then 'punycode' is śublimesecurity.com. | keyword | +| sublime_security.email_message.headers.reply_to.email.domain.root_domain | The root domain, including the TLD. | keyword | +| sublime_security.email_message.headers.reply_to.email.domain.sld | Second-level domain, e.g. 'windows' for the domain 'windows.net'. | keyword | +| sublime_security.email_message.headers.reply_to.email.domain.subdomain | Subdomain, e.g. 'drive' for the domain 'drive.google.com'. | keyword | +| sublime_security.email_message.headers.reply_to.email.domain.tld | The domain's top-level domain. E.g. the TLD of google.com is 'com'. | keyword | +| sublime_security.email_message.headers.reply_to.email.domain.valid | Whether the domain is valid. | boolean | +| sublime_security.email_message.headers.reply_to.email.local_part | Local-part, i.e. before the @. | keyword | +| sublime_security.email_message.headers.reply_to.email.value | Full email address. | keyword | +| sublime_security.email_message.headers.return_path.domain.domain | The fully qualified domain name (FQDN). This may not always be routable, e.g. when an email address contains a domain that is just a TLD with no SLD, e.g. foo@WIN-bar. | keyword | +| sublime_security.email_message.headers.return_path.domain.punycode | Interpreted punycode if the domain starts with xn--. For example, if 'domain' is 'xn--ublimesecurity-4xc.com' then 'punycode' is śublimesecurity.com. | keyword | +| sublime_security.email_message.headers.return_path.domain.root_domain | The root domain, including the TLD. | keyword | +| sublime_security.email_message.headers.return_path.domain.sld | Second-level domain, e.g. 'windows' for the domain 'windows.net'. | keyword | +| sublime_security.email_message.headers.return_path.domain.subdomain | Subdomain, e.g. 'drive' for the domain 'drive.google.com'. | keyword | +| sublime_security.email_message.headers.return_path.domain.tld | The domain's top-level domain. E.g. the TLD of google.com is 'com'. | keyword | +| sublime_security.email_message.headers.return_path.domain.valid | Whether the domain is valid. | boolean | +| sublime_security.email_message.headers.return_path.email | Full email address. | keyword | +| sublime_security.email_message.headers.return_path.local_part | Local-part, i.e. before the @. | keyword | +| sublime_security.email_message.headers.x_authenticated_domain.domain | The fully qualified domain name (FQDN). This may not always be routable, e.g. when an email address contains a domain that is just a TLD with no SLD, e.g. foo@WIN-bar. | keyword | +| sublime_security.email_message.headers.x_authenticated_domain.punycode | Interpreted punycode if the domain starts with xn--. For example, if 'domain' is 'xn--ublimesecurity-4xc.com' then 'punycode' is śublimesecurity.com. | keyword | +| sublime_security.email_message.headers.x_authenticated_domain.root_domain | The root domain, including the TLD. | keyword | +| sublime_security.email_message.headers.x_authenticated_domain.sld | Second-level domain, e.g. 'windows' for the domain 'windows.net'. | keyword | +| sublime_security.email_message.headers.x_authenticated_domain.subdomain | Subdomain, e.g. 'drive' for the domain 'drive.google.com'. | keyword | +| sublime_security.email_message.headers.x_authenticated_domain.tld | The domain's top-level domain. E.g. the TLD of google.com is 'com'. | keyword | +| sublime_security.email_message.headers.x_authenticated_domain.valid | Whether the domain is valid. | boolean | +| sublime_security.email_message.headers.x_authenticated_sender.domain.domain | The fully qualified domain name (FQDN). This may not always be routable, e.g. when an email address contains a domain that is just a TLD with no SLD, e.g. foo@WIN-bar. | keyword | +| sublime_security.email_message.headers.x_authenticated_sender.domain.punycode | Interpreted punycode if the domain starts with xn--. For example, if 'domain' is 'xn--ublimesecurity-4xc.com' then 'punycode' is śublimesecurity.com. | keyword | +| sublime_security.email_message.headers.x_authenticated_sender.domain.root_domain | The root domain, including the TLD. | keyword | +| sublime_security.email_message.headers.x_authenticated_sender.domain.sld | Second-level domain, e.g. 'windows' for the domain 'windows.net'. | keyword | +| sublime_security.email_message.headers.x_authenticated_sender.domain.subdomain | Subdomain, e.g. 'drive' for the domain 'drive.google.com'. | keyword | +| sublime_security.email_message.headers.x_authenticated_sender.domain.tld | The domain's top-level domain. E.g. the TLD of google.com is 'com'. | keyword | +| sublime_security.email_message.headers.x_authenticated_sender.domain.valid | Whether the domain is valid. | boolean | +| sublime_security.email_message.headers.x_authenticated_sender.email | Full email address. | keyword | +| sublime_security.email_message.headers.x_authenticated_sender.local_part | Local-part, i.e. before the @. | keyword | +| sublime_security.email_message.headers.x_client_ip.ip | The raw IP. | ip | +| sublime_security.email_message.headers.x_originating_ip.ip | The raw IP. | ip | +| sublime_security.email_message.headers.x_secure_server_account | X-SecureServer-Acct header, which represents a unique identifier associated with the sender's email account on a secure server and can be used to trace the email back to a specific account or user. | keyword | +| sublime_security.email_message.headers.x_sender.domain.domain | The fully qualified domain name (FQDN). This may not always be routable, e.g. when an email address contains a domain that is just a TLD with no SLD, e.g. foo@WIN-bar. | keyword | +| sublime_security.email_message.headers.x_sender.domain.punycode | Interpreted punycode if the domain starts with xn--. For example, if 'domain' is 'xn--ublimesecurity-4xc.com' then 'punycode' is śublimesecurity.com. | keyword | +| sublime_security.email_message.headers.x_sender.domain.root_domain | The root domain, including the TLD. | keyword | +| sublime_security.email_message.headers.x_sender.domain.sld | Second-level domain, e.g. 'windows' for the domain 'windows.net'. | keyword | +| sublime_security.email_message.headers.x_sender.domain.subdomain | Subdomain, e.g. 'drive' for the domain 'drive.google.com'. | keyword | +| sublime_security.email_message.headers.x_sender.domain.tld | The domain's top-level domain. E.g. the TLD of google.com is 'com'. | keyword | +| sublime_security.email_message.headers.x_sender.domain.valid | Whether the domain is valid. | boolean | +| sublime_security.email_message.headers.x_sender.email | Full email address. | keyword | +| sublime_security.email_message.headers.x_sender.local_part | Local-part, i.e. before the @. | keyword | +| sublime_security.email_message.mailbox.display_name | Display name. | keyword | +| sublime_security.email_message.mailbox.email.domain.domain | The fully qualified domain name (FQDN). This may not always be routable, e.g. when an email address contains a domain that is just a TLD with no SLD, e.g. foo@WIN-bar. | keyword | +| sublime_security.email_message.mailbox.email.domain.punycode | Interpreted punycode if the domain starts with xn--. For example, if 'domain' is 'xn--ublimesecurity-4xc.com' then 'punycode' is śublimesecurity.com. | keyword | +| sublime_security.email_message.mailbox.email.domain.root_domain | The root domain, including the TLD. | keyword | +| sublime_security.email_message.mailbox.email.domain.sld | Second-level domain, e.g. 'windows' for the domain 'windows.net'. | keyword | +| sublime_security.email_message.mailbox.email.domain.subdomain | Subdomain, e.g. 'drive' for the domain 'drive.google.com'. | keyword | +| sublime_security.email_message.mailbox.email.domain.tld | The domain's top-level domain. E.g. the TLD of google.com is 'com'. | keyword | +| sublime_security.email_message.mailbox.email.domain.valid | Whether the domain is valid. | boolean | +| sublime_security.email_message.mailbox.email.local_part | Local-part, i.e. before the @. | keyword | +| sublime_security.email_message.mailbox.email.value | Full email address. | keyword | +| sublime_security.email_message.meta.canonical_id | A deterministic ID, generated from metadata such as Attachments, Body, Subject, Sender and is used to group similar messages/campaigns together. | keyword | +| sublime_security.email_message.meta.created_at | Creation time of the data model. | date | +| sublime_security.email_message.meta.effective_at | Effective time of the data model, used for evaluation against lists and historical functions such as sender profiles or whois. | date | +| sublime_security.email_message.meta.id | Message ID. | keyword | +| sublime_security.email_message.recipients.bcc.display_name | Display name. | keyword | +| sublime_security.email_message.recipients.bcc.email.domain.domain | The fully qualified domain name (FQDN). This may not always be routable, e.g. when an email address contains a domain that is just a TLD with no SLD, e.g. foo@WIN-bar. | keyword | +| sublime_security.email_message.recipients.bcc.email.domain.punycode | Interpreted punycode if the domain starts with xn--. For example, if 'domain' is 'xn--ublimesecurity-4xc.com' then 'punycode' is śublimesecurity.com. | keyword | +| sublime_security.email_message.recipients.bcc.email.domain.root_domain | The root domain, including the TLD. | keyword | +| sublime_security.email_message.recipients.bcc.email.domain.sld | Second-level domain, e.g. 'windows' for the domain 'windows.net'. | keyword | +| sublime_security.email_message.recipients.bcc.email.domain.subdomain | Subdomain, e.g. 'drive' for the domain 'drive.google.com'. | keyword | +| sublime_security.email_message.recipients.bcc.email.domain.tld | The domain's top-level domain. E.g. the TLD of google.com is 'com'. | keyword | +| sublime_security.email_message.recipients.bcc.email.domain.valid | Whether the domain is valid. | boolean | +| sublime_security.email_message.recipients.bcc.email.local_part | Local-part, i.e. before the @. | keyword | +| sublime_security.email_message.recipients.bcc.email.value | Full email address. | keyword | +| sublime_security.email_message.recipients.cc.display_name | Display name. | keyword | +| sublime_security.email_message.recipients.cc.email.domain.domain | The fully qualified domain name (FQDN). This may not always be routable, e.g. when an email address contains a domain that is just a TLD with no SLD, e.g. foo@WIN-bar. | keyword | +| sublime_security.email_message.recipients.cc.email.domain.punycode | Interpreted punycode if the domain starts with xn--. For example, if 'domain' is 'xn--ublimesecurity-4xc.com' then 'punycode' is śublimesecurity.com. | keyword | +| sublime_security.email_message.recipients.cc.email.domain.root_domain | The root domain, including the TLD. | keyword | +| sublime_security.email_message.recipients.cc.email.domain.sld | Second-level domain, e.g. 'windows' for the domain 'windows.net'. | keyword | +| sublime_security.email_message.recipients.cc.email.domain.subdomain | Subdomain, e.g. 'drive' for the domain 'drive.google.com'. | keyword | +| sublime_security.email_message.recipients.cc.email.domain.tld | The domain's top-level domain. E.g. the TLD of google.com is 'com'. | keyword | +| sublime_security.email_message.recipients.cc.email.domain.valid | Whether the domain is valid. | boolean | +| sublime_security.email_message.recipients.cc.email.local_part | Local-part, i.e. before the @. | keyword | +| sublime_security.email_message.recipients.cc.email.value | Full email address. | keyword | +| sublime_security.email_message.recipients.to.display_name | Display name. | keyword | +| sublime_security.email_message.recipients.to.email.domain.domain | The fully qualified domain name (FQDN). This may not always be routable, e.g. when an email address contains a domain that is just a TLD with no SLD, e.g. foo@WIN-bar. | keyword | +| sublime_security.email_message.recipients.to.email.domain.punycode | Interpreted punycode if the domain starts with xn--. For example, if 'domain' is 'xn--ublimesecurity-4xc.com' then 'punycode' is śublimesecurity.com. | keyword | +| sublime_security.email_message.recipients.to.email.domain.root_domain | The root domain, including the TLD. | keyword | +| sublime_security.email_message.recipients.to.email.domain.sld | Second-level domain, e.g. 'windows' for the domain 'windows.net'. | keyword | +| sublime_security.email_message.recipients.to.email.domain.subdomain | Subdomain, e.g. 'drive' for the domain 'drive.google.com'. | keyword | +| sublime_security.email_message.recipients.to.email.domain.tld | The domain's top-level domain. E.g. the TLD of google.com is 'com'. | keyword | +| sublime_security.email_message.recipients.to.email.domain.valid | Whether the domain is valid. | boolean | +| sublime_security.email_message.recipients.to.email.local_part | Local-part, i.e. before the @. | keyword | +| sublime_security.email_message.recipients.to.email.value | Full email address. | keyword | +| sublime_security.email_message.sender.display_name | Display name. | keyword | +| sublime_security.email_message.sender.email.domain.domain | The fully qualified domain name (FQDN). This may not always be routable, e.g. when an email address contains a domain that is just a TLD with no SLD, e.g. foo@WIN-bar. | keyword | +| sublime_security.email_message.sender.email.domain.punycode | Interpreted punycode if the domain starts with xn--. For example, if 'domain' is 'xn--ublimesecurity-4xc.com' then 'punycode' is śublimesecurity.com. | keyword | +| sublime_security.email_message.sender.email.domain.root_domain | The root domain, including the TLD. | keyword | +| sublime_security.email_message.sender.email.domain.sld | Second-level domain, e.g. 'windows' for the domain 'windows.net'. | keyword | +| sublime_security.email_message.sender.email.domain.subdomain | Subdomain, e.g. 'drive' for the domain 'drive.google.com'. | keyword | +| sublime_security.email_message.sender.email.domain.tld | The domain's top-level domain. E.g. the TLD of google.com is 'com'. | keyword | +| sublime_security.email_message.sender.email.domain.valid | Whether the domain is valid. | boolean | +| sublime_security.email_message.sender.email.local_part | Local-part, i.e. before the @. | keyword | +| sublime_security.email_message.sender.email.value | Full email address. | keyword | +| sublime_security.email_message.subject.subject | Subject of the email. | keyword | +| sublime_security.email_message.type.inbound | Message was sent from someone outside your organization, to at least one recipient inside your organization. | boolean | +| sublime_security.email_message.type.internal | Message was sent from someone inside your organization, to at least one recipient inside your organization. Messages must be authenticated by either SPF or DKIM to be treated as internal. | boolean | +| sublime_security.email_message.type.outbound | Message was sent from someone inside your organization, to at least one recipient outside your organization. | boolean | + + +### Message Event + +This is the `message_event` dataset. + +#### Example + +An example event for `message_event` looks as following: + +```json +{ + "@timestamp": "2024-07-12T05:15:08.221Z", + "agent": { + "ephemeral_id": "384edc61-b94b-40cf-9cc6-86d5418d35e5", + "id": "5f3fcbb9-1a97-4ff3-857f-167af6664464", + "name": "docker-fleet-agent", + "type": "filebeat", + "version": "8.13.0" + }, + "data_stream": { + "dataset": "sublime_security.message_event", + "namespace": "17638", + "type": "logs" + }, + "ecs": { + "version": "8.11.0" + }, + "elastic_agent": { + "id": "5f3fcbb9-1a97-4ff3-857f-167af6664464", + "snapshot": false, + "version": "8.13.0" + }, + "email": { + "from": { + "address": [ + "bob.demo@gmail.com" + ] + }, + "subject": "Urgent: Wire transfer", + "to": { + "address": [ + "xyz@example.com", + "user12@example.com", + "user@example.com", + "leon12@example.com" + ] + } + }, + "event": { + "agent_id_status": "verified", + "category": [ + "email" + ], + "dataset": "sublime_security.message_event", + "id": "9c426680-5cdf-4283-adbd-d79ba0e52434", + "ingested": "2024-08-28T10:36:57Z", + "kind": "event", + "original": "{\"canonical_id\":\"dd97dc82731ff7e82edfccaef59826cccd271bd4423e09d1e150ade83037cb37\",\"created_at\":\"2024-07-12T05:15:08.221838Z\",\"external_id\":\"7a2dfbeb-1310-48fc-9ed9-f480608a0306\",\"forward_recipients\":[],\"forwarded_at\":null,\"id\":\"9c426680-5cdf-4283-adbd-d79ba0e52434\",\"landed_in_spam\":false,\"mailbox\":{\"email\":\"demo@example.com\",\"external_id\":null,\"id\":\"433fe142-e2e5-4372-84ea-480279543a9b\"},\"message_source_id\":\"257982a1-f106-4c68-bc64-ff032914ed5f\",\"read_at\":null,\"recipients\":[{\"email\":\"xyz@example.com\"},{\"email\":\"user12@example.com\"},{\"email\":\"user@example.com\"},{\"email\":\"leon12@example.com\"}],\"replied_at\":null,\"sender\":{\"display_name\":\"Bob Doe\",\"email\":\"bob.demo@gmail.com\"},\"subject\":\"Urgent: Wire transfer\"}", + "type": [ + "info" + ] + }, + "input": { + "type": "cel" + }, + "observer": { + "product": "Sublime Security", + "vendor": "Sublime Security" + }, + "related": { + "user": [ + "xyz@example.com", + "user12@example.com", + "user@example.com", + "leon12@example.com", + "Bob Doe", + "bob.demo@gmail.com" + ] + }, + "source": { + "user": { + "name": "Bob Doe" + } + }, + "sublime_security": { + "message_event": { + "canonical_id": "dd97dc82731ff7e82edfccaef59826cccd271bd4423e09d1e150ade83037cb37", + "created_at": "2024-07-12T05:15:08.221Z", + "external_id": "7a2dfbeb-1310-48fc-9ed9-f480608a0306", + "id": "9c426680-5cdf-4283-adbd-d79ba0e52434", + "landed_in_spam": false, + "mailbox": { + "email": "demo@example.com", + "id": "433fe142-e2e5-4372-84ea-480279543a9b" + }, + "message_source_id": "257982a1-f106-4c68-bc64-ff032914ed5f", + "recipients": [ + { + "email": "xyz@example.com" + }, + { + "email": "user12@example.com" + }, + { + "email": "user@example.com" + }, + { + "email": "leon12@example.com" + } + ], + "sender": { + "display_name": "Bob Doe", + "email": "bob.demo@gmail.com" + }, + "subject": "Urgent: Wire transfer" + } + }, + "tags": [ + "preserve_original_event", + "preserve_duplicate_custom_fields", + "forwarded", + "sublime_security-message_event" + ] +} +``` + +**Exported fields** + +| Field | Description | Type | +|---|---|---| +| @timestamp | Event timestamp. | date | +| data_stream.dataset | Data stream dataset. | constant_keyword | +| data_stream.namespace | Data stream namespace. | constant_keyword | +| data_stream.type | Data stream type. | constant_keyword | +| event.dataset | Event dataset. | constant_keyword | +| event.module | Event module. | constant_keyword | +| input.type | Type of filebeat input. | keyword | +| log.offset | Log offset. | long | +| sublime_security.message_event.canonical_id | Canonical ID of the message. | keyword | +| sublime_security.message_event.created_at | Time this message was added to sublime_security. | date | +| sublime_security.message_event.data.flagged_rules.id | ID of the flagged rule. | keyword | +| sublime_security.message_event.data.flagged_rules.name | Name of the flagged rule. | keyword | +| sublime_security.message_event.data.flagged_rules.severity | Severity of the flagged rule. | keyword | +| sublime_security.message_event.data.flagged_rules.tags | List of tags for the flagged rule. | keyword | +| sublime_security.message_event.data.triggered_actions.id | | keyword | +| sublime_security.message_event.data.triggered_actions.name | | keyword | +| sublime_security.message_event.data.triggered_actions.type | | keyword | +| sublime_security.message_event.external_id | ID of the message in the source system (e.g., Office 365 or Google Workspace). | keyword | +| sublime_security.message_event.forward_recipients | Email addresses this message was forwarded to by the recipient mailbox. | keyword | +| sublime_security.message_event.forwarded_at | Time this message was forwarded by the recipient mailbox. A null value indicates that it has not yet been forwarded. | date | +| sublime_security.message_event.id | Message ID. | keyword | +| sublime_security.message_event.landed_in_spam | Whether the message landed in the recipient's spam folder. | boolean | +| sublime_security.message_event.mailbox.email | Mailbox email address. | keyword | +| sublime_security.message_event.mailbox.external_id | ID of the mailbox in the source system (e.g., Office 365 or Google Workspace). | keyword | +| sublime_security.message_event.mailbox.id | Mailbox ID. | keyword | +| sublime_security.message_event.message_source_id | ID of the message source of the message. | keyword | +| sublime_security.message_event.read_at | Time this message was read in the user's mailbox. A null value indicates that it has not yet been marked read. | date | +| sublime_security.message_event.recipients.email | Email address. | keyword | +| sublime_security.message_event.replied_at | Time that this message was replied to by the recipient mailbox. A null value indicates that it has not yet been replied to by the recipient. | date | +| sublime_security.message_event.sender.display_name | Display name. | keyword | +| sublime_security.message_event.sender.email | Email address. | keyword | +| sublime_security.message_event.subject | Subject of the message. | keyword | +| sublime_security.message_event.type | | keyword | + diff --git a/packages/sublime_security/img/sublime_security-audit.png b/packages/sublime_security/img/sublime_security-audit.png new file mode 100644 index 00000000000..dcd643bf146 Binary files /dev/null and b/packages/sublime_security/img/sublime_security-audit.png differ diff --git a/packages/sublime_security/img/sublime_security-dark.svg b/packages/sublime_security/img/sublime_security-dark.svg new file mode 100644 index 00000000000..5674089d495 --- /dev/null +++ b/packages/sublime_security/img/sublime_security-dark.svg @@ -0,0 +1,18 @@ + + + + + + + + + + + + + + + + + + diff --git a/packages/sublime_security/img/sublime_security-email_message.png b/packages/sublime_security/img/sublime_security-email_message.png new file mode 100644 index 00000000000..59b49b4e9ba Binary files /dev/null and b/packages/sublime_security/img/sublime_security-email_message.png differ diff --git a/packages/sublime_security/img/sublime_security-message_event.png b/packages/sublime_security/img/sublime_security-message_event.png new file mode 100644 index 00000000000..ae44c0f2109 Binary files /dev/null and b/packages/sublime_security/img/sublime_security-message_event.png differ diff --git a/packages/sublime_security/img/sublime_security.svg b/packages/sublime_security/img/sublime_security.svg new file mode 100644 index 00000000000..529a8cbe310 --- /dev/null +++ b/packages/sublime_security/img/sublime_security.svg @@ -0,0 +1,18 @@ + + + + + + + + + + + + + + + + + + diff --git a/packages/sublime_security/kibana/dashboard/sublime_security-779aade2-fbb2-425d-8647-79c2bdf2d6e0.json b/packages/sublime_security/kibana/dashboard/sublime_security-779aade2-fbb2-425d-8647-79c2bdf2d6e0.json new file mode 100644 index 00000000000..1ecd5f06a3b --- /dev/null +++ b/packages/sublime_security/kibana/dashboard/sublime_security-779aade2-fbb2-425d-8647-79c2bdf2d6e0.json @@ -0,0 +1,2328 @@ +{ + "attributes": { + "controlGroupInput": { + "chainingSystem": "HIERARCHICAL", + "controlStyle": "oneLine", + "ignoreParentSettingsJSON": { + "ignoreFilters": false, + "ignoreQuery": false, + "ignoreTimerange": false, + "ignoreValidations": false + }, + "panelsJSON": { + "43086ab7-f19e-47ae-9bd2-ac66dd3f65c2": { + "explicitInput": { + "enhancements": {}, + "fieldName": "email.direction", + "grow": true, + "id": "43086ab7-f19e-47ae-9bd2-ac66dd3f65c2", + "searchTechnique": "prefix", + "selectedOptions": [], + "title": "Email Direction", + "width": "medium" + }, + "grow": true, + "order": 1, + "type": "optionsListControl", + "width": "medium" + }, + "5bf6abc3-e8bf-43d9-8c24-9e5212675ebc": { + "explicitInput": { + "enhancements": {}, + "fieldName": "sublime_security.email_message.headers.hops.spf_details.verdict", + "grow": true, + "id": "5bf6abc3-e8bf-43d9-8c24-9e5212675ebc", + "searchTechnique": "prefix", + "title": "SPF Verdict", + "width": "medium" + }, + "grow": true, + "order": 2, + "type": "optionsListControl", + "width": "medium" + }, + "a4c5939d-b0f2-4ca2-8966-ebe67c2a2e43": { + "explicitInput": { + "enhancements": {}, + "fieldName": "sublime_security.email_message.headers.hops.authentication_results.dmarc_details.verdict", + "grow": true, + "id": "a4c5939d-b0f2-4ca2-8966-ebe67c2a2e43", + "searchTechnique": "prefix", + "title": "Dmarc Verdict", + "width": "medium" + }, + "grow": true, + "order": 3, + "type": "optionsListControl", + "width": "medium" + }, + "d1ffec2f-3d34-437f-ac80-f94931769da4": { + "explicitInput": { + "enhancements": {}, + "fieldName": "sublime_security.email_message.headers.auth_summary.dmarc.details.disposition", + "grow": true, + "id": "d1ffec2f-3d34-437f-ac80-f94931769da4", + "searchTechnique": "prefix", + "title": "Dmarc Details Dispostion", + "width": "medium" + }, + "grow": true, + "order": 0, + "type": "optionsListControl", + "width": "medium" + } + } + }, + "description": "This dashboard shows Email Message logs collected by the Sublime Security integration.", + "kibanaSavedObjectMeta": { + "searchSourceJSON": { + "filter": [ + { + "$state": { + "store": "appState" + }, + "meta": { + "alias": null, + "disabled": false, + "field": "data_stream.dataset", + "indexRefName": "kibanaSavedObjectMeta.searchSourceJSON.filter[0].meta.index", + "key": "data_stream.dataset", + "negate": false, + "params": { + "query": "sublime_security.email_message" + }, + "type": "phrase" + }, + "query": { + "match_phrase": { + "data_stream.dataset": "sublime_security.email_message" + } + } + } + ], + "query": { + "language": "kuery", + "query": "" + } + } + }, + "optionsJSON": { + "hidePanelTitles": false, + "syncColors": false, + "syncCursor": true, + "syncTooltips": false, + "useMargins": true + }, + "panelsJSON": [ + { + "embeddableConfig": { + "enhancements": {}, + "savedVis": { + "data": { + "aggs": [], + "searchSource": { + "filter": [], + "query": { + "language": "kuery", + "query": "" + } + } + }, + "description": "", + "id": "", + "params": { + "fontSize": 12, + "markdown": "**Navigation**\n\n**Sublime Security** \n\n- [Audit](#/dashboard/sublime_security-7b4299fc-2465-46c6-bc55-dba692bb2745)\n- [**Email Message (This Page)**](#/dashboard/sublime_security-779aade2-fbb2-425d-8647-79c2bdf2d6e0)\n- [Message Event](#/dashboard/sublime_security-f4f4e3ca-1993-4a55-9d87-a7029ee0f869)\n\n**Overview**\nThis dashboard provides comprehensive statistics and visualizes the ingested logs associated with email message activities from Sublime Security.\n\nThe dashboard offers a detailed analysis of email activities, presenting a timeline of email messages and distinguishing between inbound and outbound email messages. It highlights the top 10 email addresses with the most outbound activities and identifies the most spam mailer.\n\nThe dashboard also categorizes emails by source and destination domains, as well as by disposition, providing insights into their status and actions taken and also includes a count of total email messages for comprehensive monitoring.\n\n[**Integrations Page**](/app/integrations/detail/sublime_security-0.1.0/overview)", + "openLinksInNewTab": false + }, + "title": "", + "type": "markdown", + "uiState": {} + } + }, + "gridData": { + "h": 30, + "i": "48d9fa5e-fec0-4665-ab20-622135361b15", + "w": 12, + "x": 0, + "y": 0 + }, + "panelIndex": "48d9fa5e-fec0-4665-ab20-622135361b15", + "title": "Table of Contents", + "type": "visualization" + }, + { + "embeddableConfig": { + "attributes": { + "references": [ + { + "id": "logs-*", + "name": "indexpattern-datasource-layer-1b1df76e-81b7-4bb9-9f76-56b2b4f3d069", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "64a20f49-2909-4aa3-9cfa-8a6fca1059ce", + "type": "index-pattern" + } + ], + "state": { + "adHocDataViews": {}, + "datasourceStates": { + "formBased": { + "layers": { + "1b1df76e-81b7-4bb9-9f76-56b2b4f3d069": { + "columnOrder": [ + "3ff9778d-4b70-4219-8e4c-6385ef6d8aee" + ], + "columns": { + "3ff9778d-4b70-4219-8e4c-6385ef6d8aee": { + "customLabel": true, + "dataType": "number", + "isBucketed": false, + "label": "Total Email Messages", + "operationType": "count", + "params": { + "emptyAsNull": false, + "format": { + "id": "number", + "params": { + "decimals": 0 + } + } + }, + "scale": "ratio", + "sourceField": "___records___" + } + }, + "ignoreGlobalFilters": false, + "incompleteColumns": {}, + "sampling": 1 + } + } + }, + "indexpattern": { + "layers": {} + }, + "textBased": { + "layers": {} + } + }, + "filters": [ + { + "$state": { + "store": "appState" + }, + "meta": { + "alias": null, + "disabled": false, + "field": "data_stream.dataset", + "index": "64a20f49-2909-4aa3-9cfa-8a6fca1059ce", + "key": "data_stream.dataset", + "negate": false, + "params": { + "query": "sublime_security.email_message" + }, + "type": "phrase" + }, + "query": { + "match_phrase": { + "data_stream.dataset": "sublime_security.email_message" + } + } + } + ], + "internalReferences": [], + "query": { + "language": "kuery", + "query": "" + }, + "visualization": { + "color": "#6092C0", + "layerId": "1b1df76e-81b7-4bb9-9f76-56b2b4f3d069", + "layerType": "data", + "metricAccessor": "3ff9778d-4b70-4219-8e4c-6385ef6d8aee" + } + }, + "title": "", + "type": "lens", + "visualizationType": "lnsMetric" + }, + "enhancements": {}, + "hidePanelTitles": true + }, + "gridData": { + "h": 12, + "i": "97225a77-bdbe-4f64-ac69-225509f72a1b", + "w": 9, + "x": 12, + "y": 0 + }, + "panelIndex": "97225a77-bdbe-4f64-ac69-225509f72a1b", + "title": "Total Email Messages [Logs Sublime Security]", + "type": "lens" + }, + { + "embeddableConfig": { + "attributes": { + "references": [ + { + "id": "logs-*", + "name": "indexpattern-datasource-layer-5c9b0e1a-bbcb-4655-bd37-bb80acda8b41", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "57d83bcb-3c1f-4b50-85a1-414c93b7b278", + "type": "index-pattern" + } + ], + "state": { + "adHocDataViews": {}, + "datasourceStates": { + "formBased": { + "layers": { + "5c9b0e1a-bbcb-4655-bd37-bb80acda8b41": { + "columnOrder": [ + "786f13d2-c57f-4f75-9cc5-00b25767d78d" + ], + "columns": { + "786f13d2-c57f-4f75-9cc5-00b25767d78d": { + "customLabel": true, + "dataType": "number", + "filter": { + "language": "kuery", + "query": "" + }, + "isBucketed": false, + "label": "Internal Email Messages", + "operationType": "count", + "params": { + "emptyAsNull": false, + "format": { + "id": "number", + "params": { + "decimals": 0 + } + } + }, + "scale": "ratio", + "sourceField": "___records___" + } + }, + "ignoreGlobalFilters": false, + "incompleteColumns": {}, + "sampling": 1 + } + } + }, + "indexpattern": { + "layers": {} + }, + "textBased": { + "layers": {} + } + }, + "filters": [ + { + "$state": { + "store": "appState" + }, + "meta": { + "alias": null, + "disabled": false, + "index": "57d83bcb-3c1f-4b50-85a1-414c93b7b278", + "negate": false, + "params": [ + { + "meta": { + "alias": null, + "disabled": false, + "field": "data_stream.dataset", + "index": "logs-*", + "key": "data_stream.dataset", + "negate": false, + "params": { + "query": "sublime_security.email_message" + }, + "type": "phrase" + }, + "query": { + "match_phrase": { + "data_stream.dataset": "sublime_security.email_message" + } + } + }, + { + "meta": { + "alias": null, + "disabled": false, + "field": "email.direction", + "index": "logs-*", + "key": "email.direction", + "negate": false, + "params": { + "query": "internal" + }, + "type": "phrase" + }, + "query": { + "match_phrase": { + "email.direction": "internal" + } + } + } + ], + "relation": "AND", + "type": "combined" + }, + "query": {} + } + ], + "internalReferences": [], + "query": { + "language": "kuery", + "query": "" + }, + "visualization": { + "color": "#6092C0", + "layerId": "5c9b0e1a-bbcb-4655-bd37-bb80acda8b41", + "layerType": "data", + "metricAccessor": "786f13d2-c57f-4f75-9cc5-00b25767d78d" + } + }, + "title": "", + "type": "lens", + "visualizationType": "lnsMetric" + }, + "enhancements": {}, + "hidePanelTitles": true + }, + "gridData": { + "h": 12, + "i": "1519ff06-42ba-44ee-8821-fd399237ac69", + "w": 9, + "x": 21, + "y": 0 + }, + "panelIndex": "1519ff06-42ba-44ee-8821-fd399237ac69", + "title": "Internal Email Messages [Logs Sublime Security]", + "type": "lens" + }, + { + "embeddableConfig": { + "attributes": { + "references": [ + { + "id": "logs-*", + "name": "indexpattern-datasource-layer-ca695901-8302-4c90-a629-4e6b1ea9080d", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "45421d02-30f5-44cc-ae81-774547f42ec0", + "type": "index-pattern" + } + ], + "state": { + "adHocDataViews": {}, + "datasourceStates": { + "formBased": { + "layers": { + "ca695901-8302-4c90-a629-4e6b1ea9080d": { + "columnOrder": [ + "b86f9816-39f3-47e1-a404-87979069aac0" + ], + "columns": { + "b86f9816-39f3-47e1-a404-87979069aac0": { + "customLabel": true, + "dataType": "number", + "filter": { + "language": "kuery", + "query": "" + }, + "isBucketed": false, + "label": "Inbound Email Messages", + "operationType": "count", + "params": { + "emptyAsNull": false, + "format": { + "id": "number", + "params": { + "decimals": 0 + } + } + }, + "scale": "ratio", + "sourceField": "___records___" + } + }, + "ignoreGlobalFilters": false, + "incompleteColumns": {}, + "sampling": 1 + } + } + }, + "indexpattern": { + "layers": {} + }, + "textBased": { + "layers": {} + } + }, + "filters": [ + { + "$state": { + "store": "appState" + }, + "meta": { + "alias": null, + "disabled": false, + "index": "45421d02-30f5-44cc-ae81-774547f42ec0", + "negate": false, + "params": [ + { + "meta": { + "alias": null, + "disabled": false, + "field": "data_stream.dataset", + "index": "logs-*", + "key": "data_stream.dataset", + "negate": false, + "params": { + "query": "sublime_security.email_message" + }, + "type": "phrase" + }, + "query": { + "match_phrase": { + "data_stream.dataset": "sublime_security.email_message" + } + } + }, + { + "meta": { + "alias": null, + "disabled": false, + "field": "email.direction", + "index": "logs-*", + "key": "email.direction", + "negate": false, + "params": { + "query": "inbound" + }, + "type": "phrase" + }, + "query": { + "match_phrase": { + "email.direction": "inbound" + } + } + } + ], + "relation": "AND", + "type": "combined" + }, + "query": {} + } + ], + "internalReferences": [], + "query": { + "language": "kuery", + "query": "" + }, + "visualization": { + "color": "#6092C0", + "layerId": "ca695901-8302-4c90-a629-4e6b1ea9080d", + "layerType": "data", + "metricAccessor": "b86f9816-39f3-47e1-a404-87979069aac0" + } + }, + "title": "", + "type": "lens", + "visualizationType": "lnsMetric" + }, + "enhancements": {}, + "hidePanelTitles": true + }, + "gridData": { + "h": 12, + "i": "7e393a0d-438d-4317-98ff-d61974a8c500", + "w": 9, + "x": 30, + "y": 0 + }, + "panelIndex": "7e393a0d-438d-4317-98ff-d61974a8c500", + "title": "Inbound Email Messages [Logs Sublime Security]", + "type": "lens" + }, + { + "embeddableConfig": { + "attributes": { + "references": [ + { + "id": "logs-*", + "name": "indexpattern-datasource-layer-95b5f68c-40f5-4b06-9162-73d5890122cf", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "fe61cf6c-f702-436a-a660-991b069e50ba", + "type": "index-pattern" + } + ], + "state": { + "adHocDataViews": {}, + "datasourceStates": { + "formBased": { + "layers": { + "95b5f68c-40f5-4b06-9162-73d5890122cf": { + "columnOrder": [ + "0bcdf150-a601-4485-bcc0-6cfe0dd12680" + ], + "columns": { + "0bcdf150-a601-4485-bcc0-6cfe0dd12680": { + "customLabel": true, + "dataType": "number", + "filter": { + "language": "kuery", + "query": "" + }, + "isBucketed": false, + "label": "Outbound Email Messages", + "operationType": "count", + "params": { + "emptyAsNull": false, + "format": { + "id": "number", + "params": { + "decimals": 0 + } + } + }, + "scale": "ratio", + "sourceField": "___records___" + } + }, + "ignoreGlobalFilters": false, + "incompleteColumns": {}, + "sampling": 1 + } + } + }, + "indexpattern": { + "layers": {} + }, + "textBased": { + "layers": {} + } + }, + "filters": [ + { + "$state": { + "store": "appState" + }, + "meta": { + "alias": null, + "disabled": false, + "index": "fe61cf6c-f702-436a-a660-991b069e50ba", + "negate": false, + "params": [ + { + "meta": { + "alias": null, + "disabled": false, + "field": "data_stream.dataset", + "index": "logs-*", + "key": "data_stream.dataset", + "negate": false, + "params": { + "query": "sublime_security.email_message" + }, + "type": "phrase" + }, + "query": { + "match_phrase": { + "data_stream.dataset": "sublime_security.email_message" + } + } + }, + { + "meta": { + "alias": null, + "disabled": false, + "field": "email.direction", + "index": "logs-*", + "key": "email.direction", + "negate": false, + "params": { + "query": "outbound" + }, + "type": "phrase" + }, + "query": { + "match_phrase": { + "email.direction": "outbound" + } + } + } + ], + "relation": "AND", + "type": "combined" + }, + "query": {} + } + ], + "internalReferences": [], + "query": { + "language": "kuery", + "query": "" + }, + "visualization": { + "color": "#6092C0", + "layerId": "95b5f68c-40f5-4b06-9162-73d5890122cf", + "layerType": "data", + "metricAccessor": "0bcdf150-a601-4485-bcc0-6cfe0dd12680", + "showBar": false + } + }, + "title": "", + "type": "lens", + "visualizationType": "lnsMetric" + }, + "enhancements": {}, + "hidePanelTitles": true + }, + "gridData": { + "h": 12, + "i": "5467c528-4654-441e-9374-1ebcfaf73591", + "w": 9, + "x": 39, + "y": 0 + }, + "panelIndex": "5467c528-4654-441e-9374-1ebcfaf73591", + "title": "Outbound Email Messages [Logs Sublime Security]", + "type": "lens" + }, + { + "embeddableConfig": { + "attributes": { + "references": [ + { + "id": "logs-*", + "name": "indexpattern-datasource-layer-1bd10956-4d92-4260-bbdb-d92a50b18357", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "f749bdb3-59e4-4ce9-a769-54f1e305a119", + "type": "index-pattern" + } + ], + "state": { + "adHocDataViews": {}, + "datasourceStates": { + "formBased": { + "layers": { + "1bd10956-4d92-4260-bbdb-d92a50b18357": { + "columnOrder": [ + "0423508c-0238-4efc-9a45-3eeee6e058b2", + "f36803a4-aa00-433e-84a1-e18418eb55ce" + ], + "columns": { + "0423508c-0238-4efc-9a45-3eeee6e058b2": { + "dataType": "date", + "isBucketed": true, + "label": "@timestamp", + "operationType": "date_histogram", + "params": { + "dropPartials": false, + "includeEmptyRows": true, + "interval": "auto" + }, + "scale": "interval", + "sourceField": "@timestamp" + }, + "f36803a4-aa00-433e-84a1-e18418eb55ce": { + "customLabel": true, + "dataType": "number", + "isBucketed": false, + "label": "Count", + "operationType": "count", + "params": { + "emptyAsNull": false, + "format": { + "id": "number", + "params": { + "decimals": 0 + } + } + }, + "scale": "ratio", + "sourceField": "___records___" + } + }, + "ignoreGlobalFilters": false, + "incompleteColumns": {}, + "sampling": 1 + } + } + }, + "indexpattern": { + "layers": {} + }, + "textBased": { + "layers": {} + } + }, + "filters": [ + { + "$state": { + "store": "appState" + }, + "meta": { + "alias": null, + "disabled": false, + "field": "data_stream.dataset", + "index": "f749bdb3-59e4-4ce9-a769-54f1e305a119", + "key": "data_stream.dataset", + "negate": false, + "params": { + "query": "sublime_security.email_message" + }, + "type": "phrase" + }, + "query": { + "match_phrase": { + "data_stream.dataset": "sublime_security.email_message" + } + } + } + ], + "internalReferences": [], + "query": { + "language": "kuery", + "query": "" + }, + "visualization": { + "layers": [ + { + "accessors": [ + "f36803a4-aa00-433e-84a1-e18418eb55ce" + ], + "colorMapping": { + "assignments": [], + "colorMode": { + "type": "categorical" + }, + "paletteId": "eui_amsterdam_color_blind", + "specialAssignments": [ + { + "color": { + "type": "loop" + }, + "rule": { + "type": "other" + }, + "touched": false + } + ] + }, + "layerId": "1bd10956-4d92-4260-bbdb-d92a50b18357", + "layerType": "data", + "position": "top", + "seriesType": "line", + "showGridlines": false, + "xAccessor": "0423508c-0238-4efc-9a45-3eeee6e058b2" + } + ], + "legend": { + "isVisible": true, + "position": "right", + "shouldTruncate": false, + "showSingleSeries": false + }, + "preferredSeriesType": "line", + "title": "Empty XY chart", + "valueLabels": "hide" + } + }, + "title": "", + "type": "lens", + "visualizationType": "lnsXY" + }, + "enhancements": {} + }, + "gridData": { + "h": 18, + "i": "93a69a07-ff94-40fe-a10b-f1dc36ef519e", + "w": 36, + "x": 12, + "y": 12 + }, + "panelIndex": "93a69a07-ff94-40fe-a10b-f1dc36ef519e", + "title": "Email Messages over Time [Logs Sublime Security]", + "type": "lens" + }, + { + "embeddableConfig": { + "attributes": { + "references": [ + { + "id": "logs-*", + "name": "indexpattern-datasource-layer-dc4b152c-375f-496b-9ce2-9b4b53fa39f4", + "type": "index-pattern" + } + ], + "state": { + "adHocDataViews": {}, + "datasourceStates": { + "formBased": { + "currentIndexPatternId": "logs-*", + "layers": { + "dc4b152c-375f-496b-9ce2-9b4b53fa39f4": { + "columnOrder": [ + "2af364ac-9633-401c-b429-beffab471097", + "1f39907b-3823-4501-bfe0-f1c26dd535e5" + ], + "columns": { + "1f39907b-3823-4501-bfe0-f1c26dd535e5": { + "customLabel": true, + "dataType": "number", + "isBucketed": false, + "label": "Count", + "operationType": "count", + "params": { + "emptyAsNull": false, + "format": { + "id": "number", + "params": { + "decimals": 0 + } + } + }, + "scale": "ratio", + "sourceField": "___records___" + }, + "2af364ac-9633-401c-b429-beffab471097": { + "customLabel": true, + "dataType": "string", + "isBucketed": true, + "label": "Source Domain", + "operationType": "terms", + "params": { + "exclude": [], + "excludeIsRegex": false, + "include": [], + "includeIsRegex": false, + "missingBucket": false, + "orderBy": { + "columnId": "1f39907b-3823-4501-bfe0-f1c26dd535e5", + "type": "column" + }, + "orderDirection": "desc", + "otherBucket": true, + "parentFormat": { + "id": "terms" + }, + "size": 5 + }, + "scale": "ordinal", + "sourceField": "source.domain" + } + }, + "ignoreGlobalFilters": false, + "incompleteColumns": {}, + "indexPatternId": "logs-*", + "sampling": 1 + } + } + }, + "indexpattern": { + "layers": {} + }, + "textBased": { + "layers": {} + } + }, + "filters": [ + { + "$state": { + "store": "appState" + }, + "meta": { + "alias": null, + "disabled": false, + "field": "data_stream.dataset", + "index": "dbb798d0-0156-400c-a1da-956a00e0367d", + "key": "data_stream.dataset", + "negate": false, + "params": { + "query": "sublime_security.email_message" + }, + "type": "phrase" + }, + "query": { + "match_phrase": { + "data_stream.dataset": "sublime_security.email_message" + } + } + } + ], + "internalReferences": [], + "query": { + "language": "kuery", + "query": "" + }, + "visualization": { + "layers": [ + { + "categoryDisplay": "default", + "colorMapping": { + "assignments": [], + "colorMode": { + "type": "categorical" + }, + "paletteId": "eui_amsterdam_color_blind", + "specialAssignments": [ + { + "color": { + "type": "loop" + }, + "rule": { + "type": "other" + }, + "touched": false + } + ] + }, + "layerId": "dc4b152c-375f-496b-9ce2-9b4b53fa39f4", + "layerType": "data", + "legendDisplay": "show", + "metrics": [ + "1f39907b-3823-4501-bfe0-f1c26dd535e5" + ], + "nestedLegend": false, + "numberDisplay": "percent", + "primaryGroups": [ + "2af364ac-9633-401c-b429-beffab471097" + ], + "truncateLegend": false + } + ], + "shape": "pie" + } + }, + "title": "", + "type": "lens", + "visualizationType": "lnsPie" + }, + "enhancements": {} + }, + "gridData": { + "h": 17, + "i": "b4444354-7a61-46f8-90c0-bd01c5fc43be", + "w": 24, + "x": 0, + "y": 30 + }, + "panelIndex": "b4444354-7a61-46f8-90c0-bd01c5fc43be", + "title": "Email Messages by Source Domain [Logs Sublime Security]", + "type": "lens" + }, + { + "embeddableConfig": { + "attributes": { + "references": [ + { + "id": "logs-*", + "name": "indexpattern-datasource-layer-f64b3670-dd51-4519-9652-22f0703c9257", + "type": "index-pattern" + } + ], + "state": { + "adHocDataViews": {}, + "datasourceStates": { + "formBased": { + "currentIndexPatternId": "logs-*", + "layers": { + "f64b3670-dd51-4519-9652-22f0703c9257": { + "columnOrder": [ + "eae9b345-165f-4ddd-a83c-dd549d071b06", + "c740764c-9432-4a98-bee4-cdd7a5683643" + ], + "columns": { + "c740764c-9432-4a98-bee4-cdd7a5683643": { + "customLabel": true, + "dataType": "number", + "isBucketed": false, + "label": "Count", + "operationType": "count", + "params": { + "emptyAsNull": false, + "format": { + "id": "number", + "params": { + "decimals": 0 + } + } + }, + "scale": "ratio", + "sourceField": "___records___" + }, + "eae9b345-165f-4ddd-a83c-dd549d071b06": { + "customLabel": true, + "dataType": "string", + "isBucketed": true, + "label": "Destination Domain", + "operationType": "terms", + "params": { + "exclude": [], + "excludeIsRegex": false, + "include": [], + "includeIsRegex": false, + "missingBucket": false, + "orderBy": { + "columnId": "c740764c-9432-4a98-bee4-cdd7a5683643", + "type": "column" + }, + "orderDirection": "desc", + "otherBucket": true, + "parentFormat": { + "id": "terms" + }, + "size": 5 + }, + "scale": "ordinal", + "sourceField": "destination.domain" + } + }, + "ignoreGlobalFilters": false, + "incompleteColumns": {}, + "indexPatternId": "logs-*", + "sampling": 1 + } + } + }, + "indexpattern": { + "layers": {} + }, + "textBased": { + "layers": {} + } + }, + "filters": [ + { + "$state": { + "store": "appState" + }, + "meta": { + "alias": null, + "disabled": false, + "field": "data_stream.dataset", + "index": "36efd7e9-b20b-49bb-9205-e68bb9dc1459", + "key": "data_stream.dataset", + "negate": false, + "params": { + "query": "sublime_security.email_message" + }, + "type": "phrase" + }, + "query": { + "match_phrase": { + "data_stream.dataset": "sublime_security.email_message" + } + } + } + ], + "internalReferences": [], + "query": { + "language": "kuery", + "query": "" + }, + "visualization": { + "layers": [ + { + "categoryDisplay": "default", + "colorMapping": { + "assignments": [], + "colorMode": { + "type": "categorical" + }, + "paletteId": "eui_amsterdam_color_blind", + "specialAssignments": [ + { + "color": { + "type": "loop" + }, + "rule": { + "type": "other" + }, + "touched": false + } + ] + }, + "layerId": "f64b3670-dd51-4519-9652-22f0703c9257", + "layerType": "data", + "legendDisplay": "show", + "metrics": [ + "c740764c-9432-4a98-bee4-cdd7a5683643" + ], + "nestedLegend": false, + "numberDisplay": "percent", + "primaryGroups": [ + "eae9b345-165f-4ddd-a83c-dd549d071b06" + ], + "truncateLegend": false + } + ], + "shape": "pie" + } + }, + "title": "", + "type": "lens", + "visualizationType": "lnsPie" + }, + "enhancements": {} + }, + "gridData": { + "h": 17, + "i": "cc96128b-4e7a-4f26-b5f7-1560fe66b526", + "w": 24, + "x": 24, + "y": 30 + }, + "panelIndex": "cc96128b-4e7a-4f26-b5f7-1560fe66b526", + "title": "Email Messages by Destination Domain [Logs Sublime Security]", + "type": "lens" + }, + { + "embeddableConfig": { + "attributes": { + "references": [ + { + "id": "logs-*", + "name": "indexpattern-datasource-layer-b5c8a603-1e75-4002-b8f7-480a0bff5011", + "type": "index-pattern" + } + ], + "state": { + "adHocDataViews": {}, + "datasourceStates": { + "formBased": { + "layers": { + "b5c8a603-1e75-4002-b8f7-480a0bff5011": { + "columnOrder": [ + "0456d391-905b-4277-8376-5246350e65b8", + "b3337a00-7c63-4b51-85ef-0574cfe884a7" + ], + "columns": { + "0456d391-905b-4277-8376-5246350e65b8": { + "customLabel": true, + "dataType": "string", + "isBucketed": true, + "label": "Dmark Details Disposition", + "operationType": "terms", + "params": { + "exclude": [], + "excludeIsRegex": false, + "include": [], + "includeIsRegex": false, + "missingBucket": false, + "orderBy": { + "columnId": "b3337a00-7c63-4b51-85ef-0574cfe884a7", + "type": "column" + }, + "orderDirection": "desc", + "otherBucket": true, + "parentFormat": { + "id": "terms" + }, + "size": 5 + }, + "scale": "ordinal", + "sourceField": "sublime_security.email_message.headers.auth_summary.dmarc.details.disposition" + }, + "b3337a00-7c63-4b51-85ef-0574cfe884a7": { + "customLabel": true, + "dataType": "number", + "isBucketed": false, + "label": "Count", + "operationType": "count", + "params": { + "emptyAsNull": false, + "format": { + "id": "number", + "params": { + "decimals": 0 + } + } + }, + "scale": "ratio", + "sourceField": "___records___" + } + }, + "ignoreGlobalFilters": false, + "incompleteColumns": {}, + "sampling": 1 + } + } + }, + "indexpattern": { + "layers": {} + }, + "textBased": { + "layers": {} + } + }, + "filters": [], + "internalReferences": [], + "query": { + "language": "kuery", + "query": "" + }, + "visualization": { + "layers": [ + { + "categoryDisplay": "default", + "colorMapping": { + "assignments": [], + "colorMode": { + "type": "categorical" + }, + "paletteId": "eui_amsterdam_color_blind", + "specialAssignments": [ + { + "color": { + "type": "loop" + }, + "rule": { + "type": "other" + }, + "touched": false + } + ] + }, + "layerId": "b5c8a603-1e75-4002-b8f7-480a0bff5011", + "layerType": "data", + "legendDisplay": "show", + "metrics": [ + "b3337a00-7c63-4b51-85ef-0574cfe884a7" + ], + "nestedLegend": false, + "numberDisplay": "percent", + "primaryGroups": [ + "0456d391-905b-4277-8376-5246350e65b8" + ], + "truncateLegend": false + } + ], + "shape": "pie" + } + }, + "title": "", + "type": "lens", + "visualizationType": "lnsPie" + }, + "enhancements": {} + }, + "gridData": { + "h": 18, + "i": "04ef6999-c466-423b-b1a9-b210d3b5233f", + "w": 24, + "x": 0, + "y": 47 + }, + "panelIndex": "04ef6999-c466-423b-b1a9-b210d3b5233f", + "title": "Email Messages by Disposition [Logs Sublime Security]", + "type": "lens" + }, + { + "embeddableConfig": { + "attributes": { + "references": [ + { + "id": "logs-*", + "name": "indexpattern-datasource-layer-3d92b511-53dd-4ef6-bb96-5b2d07f1632a", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "f4396f82-2fd0-4edc-8c5e-7a16340ffd4e", + "type": "index-pattern" + } + ], + "state": { + "adHocDataViews": {}, + "datasourceStates": { + "formBased": { + "layers": { + "3d92b511-53dd-4ef6-bb96-5b2d07f1632a": { + "columnOrder": [ + "eb5273b6-352a-4b1d-9128-d11c3e15425a", + "9076d461-5010-4ad8-9fa1-e3fccfa33d9c" + ], + "columns": { + "9076d461-5010-4ad8-9fa1-e3fccfa33d9c": { + "customLabel": true, + "dataType": "number", + "filter": { + "language": "kuery", + "query": "" + }, + "isBucketed": false, + "label": "Count", + "operationType": "count", + "params": { + "emptyAsNull": false, + "format": { + "id": "number", + "params": { + "decimals": 0 + } + } + }, + "scale": "ratio", + "sourceField": "___records___" + }, + "eb5273b6-352a-4b1d-9128-d11c3e15425a": { + "customLabel": true, + "dataType": "string", + "isBucketed": true, + "label": "Sender Email", + "operationType": "terms", + "params": { + "exclude": [], + "excludeIsRegex": false, + "include": [], + "includeIsRegex": false, + "missingBucket": false, + "orderBy": { + "columnId": "9076d461-5010-4ad8-9fa1-e3fccfa33d9c", + "type": "column" + }, + "orderDirection": "desc", + "otherBucket": false, + "parentFormat": { + "id": "terms" + }, + "size": 10 + }, + "scale": "ordinal", + "sourceField": "email.from.address" + } + }, + "ignoreGlobalFilters": false, + "incompleteColumns": {}, + "sampling": 1 + } + } + }, + "indexpattern": { + "layers": {} + }, + "textBased": { + "layers": {} + } + }, + "filters": [ + { + "$state": { + "store": "appState" + }, + "meta": { + "alias": null, + "disabled": false, + "index": "f4396f82-2fd0-4edc-8c5e-7a16340ffd4e", + "negate": false, + "params": [ + { + "meta": { + "alias": null, + "disabled": false, + "field": "data_stream.dataset", + "index": "logs-*", + "key": "data_stream.dataset", + "negate": false, + "params": { + "query": "sublime_security.email_message" + }, + "type": "phrase" + }, + "query": { + "match_phrase": { + "data_stream.dataset": "sublime_security.email_message" + } + } + }, + { + "$state": { + "store": "appState" + }, + "meta": { + "alias": null, + "disabled": false, + "index": "logs-*", + "negate": false, + "params": [ + { + "meta": { + "alias": null, + "disabled": false, + "field": "sublime_security.email_message.external.spam", + "index": "logs-*", + "key": "sublime_security.email_message.external.spam", + "negate": false, + "params": { + "query": true + }, + "type": "phrase" + }, + "query": { + "match_phrase": { + "sublime_security.email_message.external.spam": true + } + } + }, + { + "meta": { + "alias": null, + "disabled": false, + "field": "sublime_security.email_message.external.spam_folder", + "index": "logs-*", + "key": "sublime_security.email_message.external.spam_folder", + "negate": false, + "params": { + "query": true + }, + "type": "phrase" + }, + "query": { + "match_phrase": { + "sublime_security.email_message.external.spam_folder": true + } + } + } + ], + "relation": "OR", + "type": "combined" + } + } + ], + "relation": "AND", + "type": "combined" + }, + "query": {} + } + ], + "internalReferences": [], + "query": { + "language": "kuery", + "query": "" + }, + "visualization": { + "columns": [ + { + "columnId": "eb5273b6-352a-4b1d-9128-d11c3e15425a", + "isMetric": false, + "isTransposed": false + }, + { + "columnId": "9076d461-5010-4ad8-9fa1-e3fccfa33d9c", + "isMetric": true, + "isTransposed": false + } + ], + "layerId": "3d92b511-53dd-4ef6-bb96-5b2d07f1632a", + "layerType": "data" + } + }, + "title": "", + "type": "lens", + "visualizationType": "lnsDatatable" + }, + "enhancements": {} + }, + "gridData": { + "h": 18, + "i": "fb1baa20-db3e-4f20-a0cb-4972a24dbad9", + "w": 24, + "x": 24, + "y": 47 + }, + "panelIndex": "fb1baa20-db3e-4f20-a0cb-4972a24dbad9", + "title": "Top 10 Spam Mailer [Logs Sublime Security]", + "type": "lens" + }, + { + "embeddableConfig": { + "attributes": { + "references": [ + { + "id": "logs-*", + "name": "indexpattern-datasource-layer-4440cbe1-f8ad-4b79-a246-3bcd6decb793", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "91f01aca-9f7d-4de7-bf42-b7767d5b5f66", + "type": "index-pattern" + } + ], + "state": { + "adHocDataViews": {}, + "datasourceStates": { + "formBased": { + "layers": { + "4440cbe1-f8ad-4b79-a246-3bcd6decb793": { + "columnOrder": [ + "1c6cb8c9-6b25-47f0-b892-2267df078c11", + "223745a5-41f0-48c5-87d1-a4b422853319" + ], + "columns": { + "1c6cb8c9-6b25-47f0-b892-2267df078c11": { + "customLabel": true, + "dataType": "string", + "isBucketed": true, + "label": "Sender Email", + "operationType": "terms", + "params": { + "exclude": [], + "excludeIsRegex": false, + "include": [], + "includeIsRegex": false, + "missingBucket": false, + "orderBy": { + "columnId": "223745a5-41f0-48c5-87d1-a4b422853319", + "type": "column" + }, + "orderDirection": "desc", + "otherBucket": false, + "parentFormat": { + "id": "terms" + }, + "size": 10 + }, + "scale": "ordinal", + "sourceField": "email.from.address" + }, + "223745a5-41f0-48c5-87d1-a4b422853319": { + "customLabel": true, + "dataType": "number", + "filter": { + "language": "kuery", + "query": "sublime_security.email_message.type.outbound : true " + }, + "isBucketed": false, + "label": "Count", + "operationType": "count", + "params": { + "emptyAsNull": false, + "format": { + "id": "number", + "params": { + "decimals": 0 + } + } + }, + "scale": "ratio", + "sourceField": "___records___" + } + }, + "ignoreGlobalFilters": false, + "incompleteColumns": {}, + "sampling": 1 + } + } + }, + "indexpattern": { + "layers": {} + }, + "textBased": { + "layers": {} + } + }, + "filters": [ + { + "$state": { + "store": "appState" + }, + "meta": { + "alias": null, + "disabled": false, + "index": "91f01aca-9f7d-4de7-bf42-b7767d5b5f66", + "negate": false, + "params": [ + { + "meta": { + "alias": null, + "disabled": false, + "field": "data_stream.dataset", + "index": "logs-*", + "key": "data_stream.dataset", + "negate": false, + "params": { + "query": "sublime_security.email_message" + }, + "type": "phrase" + }, + "query": { + "match_phrase": { + "data_stream.dataset": "sublime_security.email_message" + } + } + }, + { + "meta": { + "alias": null, + "disabled": false, + "field": "email.direction", + "index": "logs-*", + "key": "email.direction", + "negate": false, + "params": { + "query": "outbound" + }, + "type": "phrase" + }, + "query": { + "match_phrase": { + "email.direction": "outbound" + } + } + } + ], + "relation": "AND", + "type": "combined" + }, + "query": {} + } + ], + "internalReferences": [], + "query": { + "language": "kuery", + "query": "" + }, + "visualization": { + "columns": [ + { + "columnId": "1c6cb8c9-6b25-47f0-b892-2267df078c11", + "isMetric": false, + "isTransposed": false + }, + { + "columnId": "223745a5-41f0-48c5-87d1-a4b422853319", + "isMetric": true, + "isTransposed": false + } + ], + "layerId": "4440cbe1-f8ad-4b79-a246-3bcd6decb793", + "layerType": "data" + } + }, + "title": "", + "type": "lens", + "visualizationType": "lnsDatatable" + }, + "enhancements": {} + }, + "gridData": { + "h": 18, + "i": "d9401f92-8373-450d-a9b8-97f9ff9ee644", + "w": 24, + "x": 0, + "y": 65 + }, + "panelIndex": "d9401f92-8373-450d-a9b8-97f9ff9ee644", + "title": "Top 10 Email with Most Outbound Activities [Logs Sublime Security]", + "type": "lens" + }, + { + "embeddableConfig": { + "attributes": { + "references": [ + { + "id": "logs-*", + "name": "indexpattern-datasource-layer-f37391dd-17ed-44aa-861d-2c2db250c64b", + "type": "index-pattern" + } + ], + "state": { + "adHocDataViews": {}, + "datasourceStates": { + "formBased": { + "currentIndexPatternId": "logs-*", + "layers": { + "f37391dd-17ed-44aa-861d-2c2db250c64b": { + "columnOrder": [ + "c400df35-cc79-4d2d-beee-f10e88c1f453", + "bf910af3-5b62-46a9-92a7-41ca04781544" + ], + "columns": { + "bf910af3-5b62-46a9-92a7-41ca04781544": { + "customLabel": true, + "dataType": "number", + "isBucketed": false, + "label": "Count", + "operationType": "count", + "params": { + "emptyAsNull": false, + "format": { + "id": "number", + "params": { + "decimals": 0 + } + } + }, + "scale": "ratio", + "sourceField": "___records___" + }, + "c400df35-cc79-4d2d-beee-f10e88c1f453": { + "customLabel": true, + "dataType": "ip", + "isBucketed": true, + "label": "Sender IP", + "operationType": "terms", + "params": { + "exclude": [], + "excludeIsRegex": false, + "include": [], + "includeIsRegex": false, + "missingBucket": false, + "orderBy": { + "columnId": "bf910af3-5b62-46a9-92a7-41ca04781544", + "type": "column" + }, + "orderDirection": "desc", + "otherBucket": false, + "parentFormat": { + "id": "terms" + }, + "secondaryFields": [], + "size": 10 + }, + "scale": "ordinal", + "sourceField": "source.ip" + } + }, + "ignoreGlobalFilters": false, + "incompleteColumns": {}, + "indexPatternId": "logs-*", + "sampling": 1 + } + } + }, + "indexpattern": { + "layers": {} + }, + "textBased": { + "layers": {} + } + }, + "filters": [ + { + "$state": { + "store": "appState" + }, + "meta": { + "alias": null, + "disabled": false, + "field": "data_stream.dataset", + "index": "69ae3de2-046b-4464-88b5-6a2e11c6a7b7", + "key": "data_stream.dataset", + "negate": false, + "params": { + "query": "sublime_security.email_message" + }, + "type": "phrase" + }, + "query": { + "match_phrase": { + "data_stream.dataset": "sublime_security.email_message" + } + } + } + ], + "internalReferences": [], + "query": { + "language": "kuery", + "query": "" + }, + "visualization": { + "columns": [ + { + "columnId": "c400df35-cc79-4d2d-beee-f10e88c1f453", + "isMetric": false, + "isTransposed": false + }, + { + "columnId": "bf910af3-5b62-46a9-92a7-41ca04781544", + "isMetric": true, + "isTransposed": false + } + ], + "layerId": "f37391dd-17ed-44aa-861d-2c2db250c64b", + "layerType": "data" + } + }, + "title": "", + "type": "lens", + "visualizationType": "lnsDatatable" + }, + "enhancements": {} + }, + "gridData": { + "h": 18, + "i": "81190a37-73e0-4826-b2ca-220b9ae62098", + "w": 24, + "x": 24, + "y": 65 + }, + "panelIndex": "81190a37-73e0-4826-b2ca-220b9ae62098", + "title": "Top 10 Sender IP [Logs Sublime Security]", + "type": "lens" + }, + { + "embeddableConfig": { + "attributes": { + "references": [ + { + "id": "logs-*", + "name": "indexpattern-datasource-layer-1cde8657-0264-4f71-8eb8-03ed187c3b6d", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "5a8c4cc3-6e83-4e69-a94e-354a46db1ef5", + "type": "index-pattern" + } + ], + "state": { + "adHocDataViews": {}, + "datasourceStates": { + "formBased": { + "layers": { + "1cde8657-0264-4f71-8eb8-03ed187c3b6d": { + "columnOrder": [ + "f7b4f5a9-fad6-4eb0-831c-53bf9adb8dfd", + "cbd7820a-2fe7-434b-bcfb-8598a845aaf7" + ], + "columns": { + "cbd7820a-2fe7-434b-bcfb-8598a845aaf7": { + "customLabel": true, + "dataType": "number", + "isBucketed": false, + "label": "Count", + "operationType": "count", + "params": { + "emptyAsNull": false, + "format": { + "id": "number", + "params": { + "decimals": 0 + } + } + }, + "scale": "ratio", + "sourceField": "___records___" + }, + "f7b4f5a9-fad6-4eb0-831c-53bf9adb8dfd": { + "customLabel": true, + "dataType": "string", + "isBucketed": true, + "label": "Sender Email", + "operationType": "terms", + "params": { + "exclude": [], + "excludeIsRegex": false, + "include": [], + "includeIsRegex": false, + "missingBucket": false, + "orderBy": { + "columnId": "cbd7820a-2fe7-434b-bcfb-8598a845aaf7", + "type": "column" + }, + "orderDirection": "desc", + "otherBucket": false, + "parentFormat": { + "id": "terms" + }, + "size": 10 + }, + "scale": "ordinal", + "sourceField": "email.from.address" + } + }, + "ignoreGlobalFilters": false, + "incompleteColumns": {}, + "sampling": 1 + } + } + }, + "indexpattern": { + "layers": {} + }, + "textBased": { + "layers": {} + } + }, + "filters": [ + { + "$state": { + "store": "appState" + }, + "meta": { + "alias": null, + "disabled": false, + "field": "data_stream.dataset", + "index": "5a8c4cc3-6e83-4e69-a94e-354a46db1ef5", + "key": "data_stream.dataset", + "negate": false, + "params": { + "query": "sublime_security.email_message" + }, + "type": "phrase" + }, + "query": { + "match_phrase": { + "data_stream.dataset": "sublime_security.email_message" + } + } + } + ], + "internalReferences": [], + "query": { + "language": "kuery", + "query": "" + }, + "visualization": { + "columns": [ + { + "columnId": "f7b4f5a9-fad6-4eb0-831c-53bf9adb8dfd", + "isMetric": false, + "isTransposed": false + }, + { + "columnId": "cbd7820a-2fe7-434b-bcfb-8598a845aaf7", + "isMetric": true, + "isTransposed": false + } + ], + "layerId": "1cde8657-0264-4f71-8eb8-03ed187c3b6d", + "layerType": "data" + } + }, + "title": "", + "type": "lens", + "visualizationType": "lnsDatatable" + }, + "enhancements": {} + }, + "gridData": { + "h": 19, + "i": "1773670c-1e6a-43cd-a398-a5b01befb525", + "w": 24, + "x": 0, + "y": 83 + }, + "panelIndex": "1773670c-1e6a-43cd-a398-a5b01befb525", + "title": "Top 10 Sender Email [Logs Sublime Security]", + "type": "lens" + }, + { + "embeddableConfig": { + "attributes": { + "references": [ + { + "id": "logs-*", + "name": "indexpattern-datasource-layer-0a0385bb-321f-4dd0-af0c-724bbe4160a4", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "c55d8cb4-ec46-4caa-873c-bb2c6db8a179", + "type": "index-pattern" + } + ], + "state": { + "adHocDataViews": {}, + "datasourceStates": { + "formBased": { + "layers": { + "0a0385bb-321f-4dd0-af0c-724bbe4160a4": { + "columnOrder": [ + "897e1123-5a3f-42f4-ac4b-da30e4cbad9d", + "01154c2a-b61e-47e5-9459-1507c39e9b74" + ], + "columns": { + "01154c2a-b61e-47e5-9459-1507c39e9b74": { + "customLabel": true, + "dataType": "number", + "isBucketed": false, + "label": "Count", + "operationType": "count", + "params": { + "emptyAsNull": false, + "format": { + "id": "number", + "params": { + "decimals": 0 + } + } + }, + "scale": "ratio", + "sourceField": "___records___" + }, + "897e1123-5a3f-42f4-ac4b-da30e4cbad9d": { + "customLabel": true, + "dataType": "string", + "isBucketed": true, + "label": "Recipient Email", + "operationType": "terms", + "params": { + "exclude": [], + "excludeIsRegex": false, + "include": [], + "includeIsRegex": false, + "missingBucket": false, + "orderBy": { + "columnId": "01154c2a-b61e-47e5-9459-1507c39e9b74", + "type": "column" + }, + "orderDirection": "desc", + "otherBucket": false, + "parentFormat": { + "id": "terms" + }, + "secondaryFields": [], + "size": 10 + }, + "scale": "ordinal", + "sourceField": "email.to.address" + } + }, + "ignoreGlobalFilters": false, + "incompleteColumns": {}, + "sampling": 1 + } + } + }, + "indexpattern": { + "layers": {} + }, + "textBased": { + "layers": {} + } + }, + "filters": [ + { + "$state": { + "store": "appState" + }, + "meta": { + "alias": null, + "disabled": false, + "field": "data_stream.dataset", + "index": "c55d8cb4-ec46-4caa-873c-bb2c6db8a179", + "key": "data_stream.dataset", + "negate": false, + "params": { + "query": "sublime_security.email_message" + }, + "type": "phrase" + }, + "query": { + "match_phrase": { + "data_stream.dataset": "sublime_security.email_message" + } + } + } + ], + "internalReferences": [], + "query": { + "language": "kuery", + "query": "" + }, + "visualization": { + "columns": [ + { + "columnId": "897e1123-5a3f-42f4-ac4b-da30e4cbad9d", + "isMetric": false, + "isTransposed": false + }, + { + "columnId": "01154c2a-b61e-47e5-9459-1507c39e9b74", + "isMetric": true, + "isTransposed": false + } + ], + "layerId": "0a0385bb-321f-4dd0-af0c-724bbe4160a4", + "layerType": "data" + } + }, + "title": "", + "type": "lens", + "visualizationType": "lnsDatatable" + }, + "enhancements": {} + }, + "gridData": { + "h": 19, + "i": "f35a4f10-f851-4e7e-bf64-5992a570d5df", + "w": 24, + "x": 24, + "y": 83 + }, + "panelIndex": "f35a4f10-f851-4e7e-bf64-5992a570d5df", + "title": "Top 10 Recipient Email [Logs Sublime Security]", + "type": "lens" + } + ], + "timeRestore": false, + "title": "[Logs Sublime Security] Email Message", + "version": 1 + }, + "coreMigrationVersion": "8.8.0", + "created_at": "2024-08-13T05:42:26.256Z", + "id": "sublime_security-779aade2-fbb2-425d-8647-79c2bdf2d6e0", + "managed": false, + "references": [ + { + "id": "logs-*", + "name": "kibanaSavedObjectMeta.searchSourceJSON.filter[0].meta.index", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "97225a77-bdbe-4f64-ac69-225509f72a1b:indexpattern-datasource-layer-1b1df76e-81b7-4bb9-9f76-56b2b4f3d069", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "97225a77-bdbe-4f64-ac69-225509f72a1b:64a20f49-2909-4aa3-9cfa-8a6fca1059ce", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "1519ff06-42ba-44ee-8821-fd399237ac69:indexpattern-datasource-layer-5c9b0e1a-bbcb-4655-bd37-bb80acda8b41", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "1519ff06-42ba-44ee-8821-fd399237ac69:57d83bcb-3c1f-4b50-85a1-414c93b7b278", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "7e393a0d-438d-4317-98ff-d61974a8c500:indexpattern-datasource-layer-ca695901-8302-4c90-a629-4e6b1ea9080d", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "7e393a0d-438d-4317-98ff-d61974a8c500:45421d02-30f5-44cc-ae81-774547f42ec0", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "5467c528-4654-441e-9374-1ebcfaf73591:indexpattern-datasource-layer-95b5f68c-40f5-4b06-9162-73d5890122cf", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "5467c528-4654-441e-9374-1ebcfaf73591:fe61cf6c-f702-436a-a660-991b069e50ba", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "93a69a07-ff94-40fe-a10b-f1dc36ef519e:indexpattern-datasource-layer-1bd10956-4d92-4260-bbdb-d92a50b18357", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "93a69a07-ff94-40fe-a10b-f1dc36ef519e:f749bdb3-59e4-4ce9-a769-54f1e305a119", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "b4444354-7a61-46f8-90c0-bd01c5fc43be:indexpattern-datasource-layer-dc4b152c-375f-496b-9ce2-9b4b53fa39f4", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "cc96128b-4e7a-4f26-b5f7-1560fe66b526:indexpattern-datasource-layer-f64b3670-dd51-4519-9652-22f0703c9257", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "04ef6999-c466-423b-b1a9-b210d3b5233f:indexpattern-datasource-layer-b5c8a603-1e75-4002-b8f7-480a0bff5011", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "fb1baa20-db3e-4f20-a0cb-4972a24dbad9:indexpattern-datasource-layer-3d92b511-53dd-4ef6-bb96-5b2d07f1632a", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "fb1baa20-db3e-4f20-a0cb-4972a24dbad9:f4396f82-2fd0-4edc-8c5e-7a16340ffd4e", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "d9401f92-8373-450d-a9b8-97f9ff9ee644:indexpattern-datasource-layer-4440cbe1-f8ad-4b79-a246-3bcd6decb793", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "d9401f92-8373-450d-a9b8-97f9ff9ee644:91f01aca-9f7d-4de7-bf42-b7767d5b5f66", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "81190a37-73e0-4826-b2ca-220b9ae62098:indexpattern-datasource-layer-f37391dd-17ed-44aa-861d-2c2db250c64b", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "1773670c-1e6a-43cd-a398-a5b01befb525:indexpattern-datasource-layer-1cde8657-0264-4f71-8eb8-03ed187c3b6d", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "1773670c-1e6a-43cd-a398-a5b01befb525:5a8c4cc3-6e83-4e69-a94e-354a46db1ef5", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "f35a4f10-f851-4e7e-bf64-5992a570d5df:indexpattern-datasource-layer-0a0385bb-321f-4dd0-af0c-724bbe4160a4", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "f35a4f10-f851-4e7e-bf64-5992a570d5df:c55d8cb4-ec46-4caa-873c-bb2c6db8a179", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "controlGroup_43086ab7-f19e-47ae-9bd2-ac66dd3f65c2:optionsListDataView", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "controlGroup_5bf6abc3-e8bf-43d9-8c24-9e5212675ebc:optionsListDataView", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "controlGroup_a4c5939d-b0f2-4ca2-8966-ebe67c2a2e43:optionsListDataView", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "controlGroup_d1ffec2f-3d34-437f-ac80-f94931769da4:optionsListDataView", + "type": "index-pattern" + } + ], + "type": "dashboard", + "typeMigrationVersion": "8.9.0" +} \ No newline at end of file diff --git a/packages/sublime_security/kibana/dashboard/sublime_security-7b4299fc-2465-46c6-bc55-dba692bb2745.json b/packages/sublime_security/kibana/dashboard/sublime_security-7b4299fc-2465-46c6-bc55-dba692bb2745.json new file mode 100644 index 00000000000..5b37128e13f --- /dev/null +++ b/packages/sublime_security/kibana/dashboard/sublime_security-7b4299fc-2465-46c6-bc55-dba692bb2745.json @@ -0,0 +1,1152 @@ +{ + "attributes": { + "controlGroupInput": { + "chainingSystem": "HIERARCHICAL", + "controlStyle": "oneLine", + "ignoreParentSettingsJSON": { + "ignoreFilters": false, + "ignoreQuery": false, + "ignoreTimerange": false, + "ignoreValidations": false + }, + "panelsJSON": { + "24cd2b90-4b2c-4e94-a80a-2294e2a1a6c5": { + "explicitInput": { + "enhancements": {}, + "fieldName": "sublime_security.audit.type", + "grow": true, + "id": "24cd2b90-4b2c-4e94-a80a-2294e2a1a6c5", + "searchTechnique": "prefix", + "title": "Audit Type", + "width": "medium" + }, + "grow": true, + "order": 1, + "type": "optionsListControl", + "width": "medium" + }, + "5a465a27-c8c0-441e-82b1-87e5f8cc8901": { + "explicitInput": { + "enhancements": {}, + "fieldName": "sublime_security.audit.data.request.authentication_method", + "grow": true, + "id": "5a465a27-c8c0-441e-82b1-87e5f8cc8901", + "searchTechnique": "prefix", + "title": "Authentication Method", + "width": "medium" + }, + "grow": true, + "order": 2, + "type": "optionsListControl", + "width": "medium" + }, + "b06a8e14-3fe4-41c9-b8e2-a14e575a9def": { + "explicitInput": { + "enhancements": {}, + "fieldName": "user.roles", + "grow": true, + "id": "b06a8e14-3fe4-41c9-b8e2-a14e575a9def", + "searchTechnique": "prefix", + "title": "User Roles", + "width": "medium" + }, + "grow": true, + "order": 0, + "type": "optionsListControl", + "width": "medium" + } + } + }, + "description": "This dashboard shows Audit logs collected by the Sublime Security integration.", + "kibanaSavedObjectMeta": { + "searchSourceJSON": { + "filter": [ + { + "$state": { + "store": "appState" + }, + "meta": { + "alias": null, + "disabled": false, + "field": "data_stream.dataset", + "indexRefName": "kibanaSavedObjectMeta.searchSourceJSON.filter[0].meta.index", + "key": "data_stream.dataset", + "negate": false, + "params": { + "query": "sublime_security.audit" + }, + "type": "phrase" + }, + "query": { + "match_phrase": { + "data_stream.dataset": "sublime_security.audit" + } + } + } + ], + "query": { + "language": "kuery", + "query": "" + } + } + }, + "optionsJSON": { + "hidePanelTitles": false, + "syncColors": false, + "syncCursor": true, + "syncTooltips": false, + "useMargins": true + }, + "panelsJSON": [ + { + "embeddableConfig": { + "enhancements": {}, + "savedVis": { + "data": { + "aggs": [], + "searchSource": { + "filter": [], + "query": { + "language": "kuery", + "query": "" + } + } + }, + "description": "", + "id": "", + "params": { + "fontSize": 12, + "markdown": "**Navigation**\n\n**Sublime Security** \n\n- [**Audit (This Page)**](#/dashboard/sublime_security-7b4299fc-2465-46c6-bc55-dba692bb2745)\n- [Email Message](#/dashboard/sublime_security-779aade2-fbb2-425d-8647-79c2bdf2d6e0)\n- [Message Event](#/dashboard/sublime_security-f4f4e3ca-1993-4a55-9d87-a7029ee0f869)\n\n**Overview**\nThis dashboard is designed to offer comprehensive statistics and display the ingested logs related to audit activities from Sublime Security.\n\nThe dashboard provides an overview of events over time and highlights the top 10 source IPs and email addresses involved in audit activities.\n\nIt shows a breakdown of activities by user roles, event types, and authentication methods.\n\n[**Integrations Page**](/app/integrations/detail/sublime_security-0.1.0/overview)", + "openLinksInNewTab": false + }, + "title": "", + "type": "markdown", + "uiState": {} + } + }, + "gridData": { + "h": 22, + "i": "133f91e7-0037-4a1b-81f8-7e5051a5817c", + "w": 13, + "x": 0, + "y": 0 + }, + "panelIndex": "133f91e7-0037-4a1b-81f8-7e5051a5817c", + "title": "Table of Contents", + "type": "visualization" + }, + { + "embeddableConfig": { + "attributes": { + "references": [ + { + "id": "logs-*", + "name": "indexpattern-datasource-layer-22b33234-fedb-4c93-9153-1ba964554623", + "type": "index-pattern" + } + ], + "state": { + "adHocDataViews": {}, + "datasourceStates": { + "formBased": { + "currentIndexPatternId": "logs-*", + "layers": { + "22b33234-fedb-4c93-9153-1ba964554623": { + "columnOrder": [ + "98955139-3224-457f-8619-4a493f0d278b", + "80a082fe-2d98-44cf-afd9-5e6e221c921a" + ], + "columns": { + "80a082fe-2d98-44cf-afd9-5e6e221c921a": { + "customLabel": true, + "dataType": "number", + "isBucketed": false, + "label": "Audit Events", + "operationType": "count", + "params": { + "emptyAsNull": false, + "format": { + "id": "number", + "params": { + "decimals": 0 + } + } + }, + "scale": "ratio", + "sourceField": "___records___" + }, + "98955139-3224-457f-8619-4a493f0d278b": { + "dataType": "date", + "isBucketed": true, + "label": "@timestamp", + "operationType": "date_histogram", + "params": { + "dropPartials": false, + "includeEmptyRows": true, + "interval": "auto" + }, + "scale": "interval", + "sourceField": "@timestamp" + } + }, + "ignoreGlobalFilters": false, + "incompleteColumns": {}, + "indexPatternId": "logs-*", + "sampling": 1 + } + } + }, + "indexpattern": { + "layers": {} + }, + "textBased": { + "layers": {} + } + }, + "filters": [ + { + "$state": { + "store": "appState" + }, + "meta": { + "alias": null, + "disabled": false, + "field": "data_stream.dataset", + "index": "3953c285-51c5-4b14-bd56-80725b9a20ab", + "key": "data_stream.dataset", + "negate": false, + "params": { + "query": "sublime_security.audit" + }, + "type": "phrase" + }, + "query": { + "match_phrase": { + "data_stream.dataset": "sublime_security.audit" + } + } + } + ], + "internalReferences": [], + "query": { + "language": "kuery", + "query": "" + }, + "visualization": { + "layers": [ + { + "accessors": [ + "80a082fe-2d98-44cf-afd9-5e6e221c921a" + ], + "colorMapping": { + "assignments": [], + "colorMode": { + "type": "categorical" + }, + "paletteId": "eui_amsterdam_color_blind", + "specialAssignments": [ + { + "color": { + "type": "loop" + }, + "rule": { + "type": "other" + }, + "touched": false + } + ] + }, + "layerId": "22b33234-fedb-4c93-9153-1ba964554623", + "layerType": "data", + "position": "top", + "seriesType": "line", + "showGridlines": false, + "xAccessor": "98955139-3224-457f-8619-4a493f0d278b" + } + ], + "legend": { + "isVisible": true, + "position": "right", + "shouldTruncate": false + }, + "preferredSeriesType": "line", + "title": "Empty XY chart", + "valueLabels": "hide" + } + }, + "title": "", + "type": "lens", + "visualizationType": "lnsXY" + }, + "enhancements": {} + }, + "gridData": { + "h": 22, + "i": "a8d0e399-96d5-4b1e-b28b-3e49bfdc0662", + "w": 35, + "x": 13, + "y": 0 + }, + "panelIndex": "a8d0e399-96d5-4b1e-b28b-3e49bfdc0662", + "title": "Audit Events over Time [Logs Sublime Security]", + "type": "lens" + }, + { + "embeddableConfig": { + "attributes": { + "references": [ + { + "id": "logs-*", + "name": "indexpattern-datasource-layer-f7f61691-b369-4e0e-b4c6-149c1d0772c9", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "0b300a06-5304-4410-8892-012ab6a46c00", + "type": "index-pattern" + } + ], + "state": { + "adHocDataViews": {}, + "datasourceStates": { + "formBased": { + "layers": { + "f7f61691-b369-4e0e-b4c6-149c1d0772c9": { + "columnOrder": [ + "7002281b-9f9e-4dbd-8496-59adadf03dc5", + "c452b748-c087-4016-9aff-65c1aadddd0d" + ], + "columns": { + "7002281b-9f9e-4dbd-8496-59adadf03dc5": { + "customLabel": true, + "dataType": "ip", + "isBucketed": true, + "label": "Source IP", + "operationType": "terms", + "params": { + "exclude": [], + "excludeIsRegex": false, + "include": [], + "includeIsRegex": false, + "missingBucket": false, + "orderBy": { + "columnId": "c452b748-c087-4016-9aff-65c1aadddd0d", + "type": "column" + }, + "orderDirection": "desc", + "otherBucket": false, + "parentFormat": { + "id": "terms" + }, + "size": 10 + }, + "scale": "ordinal", + "sourceField": "source.ip" + }, + "c452b748-c087-4016-9aff-65c1aadddd0d": { + "customLabel": true, + "dataType": "number", + "isBucketed": false, + "label": "Count", + "operationType": "count", + "params": { + "emptyAsNull": false, + "format": { + "id": "number", + "params": { + "decimals": 0 + } + } + }, + "scale": "ratio", + "sourceField": "___records___" + } + }, + "ignoreGlobalFilters": false, + "incompleteColumns": {}, + "sampling": 1 + } + } + }, + "indexpattern": { + "layers": {} + }, + "textBased": { + "layers": {} + } + }, + "filters": [ + { + "$state": { + "store": "appState" + }, + "meta": { + "alias": null, + "disabled": false, + "field": "data_stream.dataset", + "index": "0b300a06-5304-4410-8892-012ab6a46c00", + "key": "data_stream.dataset", + "negate": false, + "params": { + "query": "sublime_security.audit" + }, + "type": "phrase" + }, + "query": { + "match_phrase": { + "data_stream.dataset": "sublime_security.audit" + } + } + } + ], + "internalReferences": [], + "query": { + "language": "kuery", + "query": "" + }, + "visualization": { + "columns": [ + { + "columnId": "7002281b-9f9e-4dbd-8496-59adadf03dc5", + "isMetric": false, + "isTransposed": false + }, + { + "columnId": "c452b748-c087-4016-9aff-65c1aadddd0d", + "isMetric": true, + "isTransposed": false + } + ], + "layerId": "f7f61691-b369-4e0e-b4c6-149c1d0772c9", + "layerType": "data" + } + }, + "title": "", + "type": "lens", + "visualizationType": "lnsDatatable" + }, + "enhancements": {} + }, + "gridData": { + "h": 18, + "i": "b1a074bd-4526-4525-bd17-2a1349e13895", + "w": 24, + "x": 0, + "y": 22 + }, + "panelIndex": "b1a074bd-4526-4525-bd17-2a1349e13895", + "title": "Top 10 Source IP [Logs Sublime Security]", + "type": "lens" + }, + { + "embeddableConfig": { + "attributes": { + "references": [ + { + "id": "logs-*", + "name": "indexpattern-datasource-layer-a33924dd-4fcc-41f1-bd0c-6e0502ad206a", + "type": "index-pattern" + } + ], + "state": { + "adHocDataViews": {}, + "datasourceStates": { + "formBased": { + "currentIndexPatternId": "logs-*", + "layers": { + "a33924dd-4fcc-41f1-bd0c-6e0502ad206a": { + "columnOrder": [ + "0b39324a-db1e-4586-af21-669070ab6c41", + "f09c81c8-8251-4d9f-bf27-7608f988e8a6" + ], + "columns": { + "0b39324a-db1e-4586-af21-669070ab6c41": { + "customLabel": true, + "dataType": "string", + "isBucketed": true, + "label": "Auditor Email", + "operationType": "terms", + "params": { + "exclude": [], + "excludeIsRegex": false, + "include": [], + "includeIsRegex": false, + "missingBucket": false, + "orderBy": { + "columnId": "f09c81c8-8251-4d9f-bf27-7608f988e8a6", + "type": "column" + }, + "orderDirection": "desc", + "otherBucket": false, + "parentFormat": { + "id": "terms" + }, + "secondaryFields": [], + "size": 10 + }, + "scale": "ordinal", + "sourceField": "user.email" + }, + "f09c81c8-8251-4d9f-bf27-7608f988e8a6": { + "customLabel": true, + "dataType": "number", + "isBucketed": false, + "label": "Count", + "operationType": "count", + "params": { + "emptyAsNull": false, + "format": { + "id": "number", + "params": { + "decimals": 0 + } + } + }, + "scale": "ratio", + "sourceField": "___records___" + } + }, + "ignoreGlobalFilters": false, + "incompleteColumns": {}, + "indexPatternId": "logs-*", + "sampling": 1 + } + } + }, + "indexpattern": { + "layers": {} + }, + "textBased": { + "layers": {} + } + }, + "filters": [ + { + "$state": { + "store": "appState" + }, + "meta": { + "alias": null, + "disabled": false, + "field": "data_stream.dataset", + "index": "4a911452-0b62-4339-9a1e-6d2a4e1a4e1a", + "key": "data_stream.dataset", + "negate": false, + "params": { + "query": "sublime_security.audit" + }, + "type": "phrase" + }, + "query": { + "match_phrase": { + "data_stream.dataset": "sublime_security.audit" + } + } + } + ], + "internalReferences": [], + "query": { + "language": "kuery", + "query": "" + }, + "visualization": { + "columns": [ + { + "columnId": "0b39324a-db1e-4586-af21-669070ab6c41", + "isMetric": false, + "isTransposed": false + }, + { + "columnId": "f09c81c8-8251-4d9f-bf27-7608f988e8a6", + "isMetric": true, + "isTransposed": false + } + ], + "layerId": "a33924dd-4fcc-41f1-bd0c-6e0502ad206a", + "layerType": "data" + } + }, + "title": "", + "type": "lens", + "visualizationType": "lnsDatatable" + }, + "enhancements": {} + }, + "gridData": { + "h": 18, + "i": "d1800904-2ffe-48fc-9456-e0356adbf543", + "w": 24, + "x": 24, + "y": 22 + }, + "panelIndex": "d1800904-2ffe-48fc-9456-e0356adbf543", + "title": "Top 10 Email Address [Logs Sublime Security]", + "type": "lens" + }, + { + "embeddableConfig": { + "attributes": { + "references": [ + { + "id": "logs-*", + "name": "indexpattern-datasource-layer-47ac87f6-885e-42ca-8b55-425c2ee715a8", + "type": "index-pattern" + } + ], + "state": { + "adHocDataViews": {}, + "datasourceStates": { + "formBased": { + "currentIndexPatternId": "logs-*", + "layers": { + "47ac87f6-885e-42ca-8b55-425c2ee715a8": { + "columnOrder": [ + "1ab3bcdf-93f4-45b0-99b7-6a9d13295df9", + "dbec7c8d-b7bf-4c8f-9f32-061f737f80ee" + ], + "columns": { + "1ab3bcdf-93f4-45b0-99b7-6a9d13295df9": { + "customLabel": true, + "dataType": "string", + "isBucketed": true, + "label": "User Roles", + "operationType": "terms", + "params": { + "exclude": [], + "excludeIsRegex": false, + "include": [], + "includeIsRegex": false, + "missingBucket": false, + "orderBy": { + "columnId": "dbec7c8d-b7bf-4c8f-9f32-061f737f80ee", + "type": "column" + }, + "orderDirection": "desc", + "otherBucket": true, + "parentFormat": { + "id": "terms" + }, + "size": 5 + }, + "scale": "ordinal", + "sourceField": "user.roles" + }, + "dbec7c8d-b7bf-4c8f-9f32-061f737f80ee": { + "customLabel": true, + "dataType": "number", + "isBucketed": false, + "label": "Count", + "operationType": "count", + "params": { + "emptyAsNull": false, + "format": { + "id": "number", + "params": { + "decimals": 0 + } + } + }, + "scale": "ratio", + "sourceField": "___records___" + } + }, + "ignoreGlobalFilters": false, + "incompleteColumns": {}, + "indexPatternId": "logs-*", + "sampling": 1 + } + } + }, + "indexpattern": { + "layers": {} + }, + "textBased": { + "layers": {} + } + }, + "filters": [ + { + "$state": { + "store": "appState" + }, + "meta": { + "alias": null, + "disabled": false, + "field": "data_stream.dataset", + "index": "102cc924-8d89-45b0-abc0-0e4d7792f9d4", + "key": "data_stream.dataset", + "negate": false, + "params": { + "query": "sublime_security.audit" + }, + "type": "phrase" + }, + "query": { + "match_phrase": { + "data_stream.dataset": "sublime_security.audit" + } + } + } + ], + "internalReferences": [], + "query": { + "language": "kuery", + "query": "" + }, + "visualization": { + "layers": [ + { + "categoryDisplay": "default", + "colorMapping": { + "assignments": [], + "colorMode": { + "type": "categorical" + }, + "paletteId": "eui_amsterdam_color_blind", + "specialAssignments": [ + { + "color": { + "type": "loop" + }, + "rule": { + "type": "other" + }, + "touched": false + } + ] + }, + "layerId": "47ac87f6-885e-42ca-8b55-425c2ee715a8", + "layerType": "data", + "legendDisplay": "show", + "metrics": [ + "dbec7c8d-b7bf-4c8f-9f32-061f737f80ee" + ], + "nestedLegend": false, + "numberDisplay": "percent", + "primaryGroups": [ + "1ab3bcdf-93f4-45b0-99b7-6a9d13295df9" + ], + "truncateLegend": false + } + ], + "shape": "pie" + } + }, + "title": "", + "type": "lens", + "visualizationType": "lnsPie" + }, + "enhancements": {} + }, + "gridData": { + "h": 16, + "i": "5cf8e991-5cdf-408a-9a94-62c6e7c7cdb5", + "w": 24, + "x": 0, + "y": 40 + }, + "panelIndex": "5cf8e991-5cdf-408a-9a94-62c6e7c7cdb5", + "title": "Audit Events by Roles [Logs Sublime Security]", + "type": "lens" + }, + { + "embeddableConfig": { + "attributes": { + "references": [ + { + "id": "logs-*", + "name": "indexpattern-datasource-layer-5c679708-1962-4585-abaf-7c09dfe3b043", + "type": "index-pattern" + } + ], + "state": { + "adHocDataViews": {}, + "datasourceStates": { + "formBased": { + "currentIndexPatternId": "logs-*", + "layers": { + "5c679708-1962-4585-abaf-7c09dfe3b043": { + "columnOrder": [ + "d5de3929-4ee3-4f7b-87f2-b5a0e3205235", + "5c8a266e-853f-4d14-888b-f08bf8f5cf45" + ], + "columns": { + "5c8a266e-853f-4d14-888b-f08bf8f5cf45": { + "customLabel": true, + "dataType": "number", + "isBucketed": false, + "label": "Count", + "operationType": "count", + "params": { + "emptyAsNull": false, + "format": { + "id": "number", + "params": { + "decimals": 0 + } + } + }, + "scale": "ratio", + "sourceField": "___records___" + }, + "d5de3929-4ee3-4f7b-87f2-b5a0e3205235": { + "customLabel": true, + "dataType": "string", + "isBucketed": true, + "label": "Authentication Method", + "operationType": "terms", + "params": { + "exclude": [], + "excludeIsRegex": false, + "include": [], + "includeIsRegex": false, + "missingBucket": false, + "orderBy": { + "columnId": "5c8a266e-853f-4d14-888b-f08bf8f5cf45", + "type": "column" + }, + "orderDirection": "desc", + "otherBucket": true, + "parentFormat": { + "id": "terms" + }, + "size": 5 + }, + "scale": "ordinal", + "sourceField": "sublime_security.audit.data.request.authentication_method" + } + }, + "ignoreGlobalFilters": false, + "incompleteColumns": {}, + "indexPatternId": "logs-*", + "sampling": 1 + } + } + }, + "indexpattern": { + "layers": {} + }, + "textBased": { + "layers": {} + } + }, + "filters": [], + "internalReferences": [], + "query": { + "language": "kuery", + "query": "" + }, + "visualization": { + "layers": [ + { + "categoryDisplay": "default", + "colorMapping": { + "assignments": [], + "colorMode": { + "type": "categorical" + }, + "paletteId": "eui_amsterdam_color_blind", + "specialAssignments": [ + { + "color": { + "type": "loop" + }, + "rule": { + "type": "other" + }, + "touched": false + } + ] + }, + "layerId": "5c679708-1962-4585-abaf-7c09dfe3b043", + "layerType": "data", + "legendDisplay": "show", + "metrics": [ + "5c8a266e-853f-4d14-888b-f08bf8f5cf45" + ], + "nestedLegend": false, + "numberDisplay": "percent", + "primaryGroups": [ + "d5de3929-4ee3-4f7b-87f2-b5a0e3205235" + ], + "truncateLegend": false + } + ], + "shape": "pie" + } + }, + "title": "", + "type": "lens", + "visualizationType": "lnsPie" + }, + "enhancements": {} + }, + "gridData": { + "h": 16, + "i": "697b7157-f5cd-48e6-a896-2207f593abeb", + "w": 24, + "x": 24, + "y": 40 + }, + "panelIndex": "697b7157-f5cd-48e6-a896-2207f593abeb", + "title": "Audit Events by Authentication Method [Logs Sublime Security]", + "type": "lens" + }, + { + "embeddableConfig": { + "attributes": { + "references": [ + { + "id": "logs-*", + "name": "indexpattern-datasource-layer-690c8f2b-cb21-42bb-9835-4b8beabcd8b8", + "type": "index-pattern" + } + ], + "state": { + "adHocDataViews": {}, + "datasourceStates": { + "formBased": { + "currentIndexPatternId": "logs-*", + "layers": { + "690c8f2b-cb21-42bb-9835-4b8beabcd8b8": { + "columnOrder": [ + "119c955c-609d-4cf7-a6d6-8b996ad645f3", + "bbf7cb47-b8f5-47fa-b5b3-63800fd22cf2" + ], + "columns": { + "119c955c-609d-4cf7-a6d6-8b996ad645f3": { + "customLabel": true, + "dataType": "string", + "isBucketed": true, + "label": "Audit Type", + "operationType": "terms", + "params": { + "exclude": [], + "excludeIsRegex": false, + "include": [], + "includeIsRegex": false, + "missingBucket": false, + "orderBy": { + "columnId": "bbf7cb47-b8f5-47fa-b5b3-63800fd22cf2", + "type": "column" + }, + "orderDirection": "desc", + "otherBucket": true, + "parentFormat": { + "id": "terms" + }, + "size": 10 + }, + "scale": "ordinal", + "sourceField": "sublime_security.audit.type" + }, + "bbf7cb47-b8f5-47fa-b5b3-63800fd22cf2": { + "customLabel": true, + "dataType": "number", + "isBucketed": false, + "label": "Count", + "operationType": "count", + "params": { + "emptyAsNull": false, + "format": { + "id": "number", + "params": { + "decimals": 0 + } + } + }, + "scale": "ratio", + "sourceField": "___records___" + } + }, + "ignoreGlobalFilters": false, + "incompleteColumns": {}, + "indexPatternId": "logs-*", + "sampling": 1 + } + } + }, + "indexpattern": { + "layers": {} + }, + "textBased": { + "layers": {} + } + }, + "filters": [], + "internalReferences": [], + "query": { + "language": "kuery", + "query": "" + }, + "visualization": { + "axisTitlesVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "fittingFunction": "None", + "gridlinesVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "labelsOrientation": { + "x": 0, + "yLeft": 0, + "yRight": 0 + }, + "layers": [ + { + "accessors": [ + "bbf7cb47-b8f5-47fa-b5b3-63800fd22cf2" + ], + "colorMapping": { + "assignments": [], + "colorMode": { + "type": "categorical" + }, + "paletteId": "eui_amsterdam_color_blind", + "specialAssignments": [ + { + "color": { + "type": "loop" + }, + "rule": { + "type": "other" + }, + "touched": false + } + ] + }, + "layerId": "690c8f2b-cb21-42bb-9835-4b8beabcd8b8", + "layerType": "data", + "position": "top", + "seriesType": "bar_horizontal", + "showGridlines": false, + "xAccessor": "119c955c-609d-4cf7-a6d6-8b996ad645f3" + } + ], + "legend": { + "isVisible": true, + "position": "right", + "shouldTruncate": false, + "showSingleSeries": false + }, + "preferredSeriesType": "bar_horizontal", + "tickLabelsVisibilitySettings": { + "x": true, + "yLeft": true, + "yRight": true + }, + "valueLabels": "hide" + } + }, + "title": "", + "type": "lens", + "visualizationType": "lnsXY" + }, + "enhancements": {} + }, + "gridData": { + "h": 20, + "i": "368a354d-a2a5-4502-aba2-bc443d0fd80a", + "w": 48, + "x": 0, + "y": 56 + }, + "panelIndex": "368a354d-a2a5-4502-aba2-bc443d0fd80a", + "title": "Audit Events by Types [Logs Sublime Security]", + "type": "lens" + }, + { + "embeddableConfig": { + "enhancements": {}, + "rowsPerPage": 100 + }, + "gridData": { + "h": 19, + "i": "e18f7396-98a9-48a7-b7cb-64416715600f", + "w": 48, + "x": 0, + "y": 76 + }, + "panelIndex": "e18f7396-98a9-48a7-b7cb-64416715600f", + "panelRefName": "panel_e18f7396-98a9-48a7-b7cb-64416715600f", + "title": "Audit Event Essential Details [Logs Sublime Security]", + "type": "search" + } + ], + "timeRestore": false, + "title": "[Logs Sublime Security] Audit", + "version": 1 + }, + "coreMigrationVersion": "8.8.0", + "created_at": "2024-08-13T05:37:29.744Z", + "id": "sublime_security-7b4299fc-2465-46c6-bc55-dba692bb2745", + "managed": false, + "references": [ + { + "id": "logs-*", + "name": "kibanaSavedObjectMeta.searchSourceJSON.filter[0].meta.index", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "a8d0e399-96d5-4b1e-b28b-3e49bfdc0662:indexpattern-datasource-layer-22b33234-fedb-4c93-9153-1ba964554623", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "b1a074bd-4526-4525-bd17-2a1349e13895:indexpattern-datasource-layer-f7f61691-b369-4e0e-b4c6-149c1d0772c9", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "b1a074bd-4526-4525-bd17-2a1349e13895:0b300a06-5304-4410-8892-012ab6a46c00", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "d1800904-2ffe-48fc-9456-e0356adbf543:indexpattern-datasource-layer-a33924dd-4fcc-41f1-bd0c-6e0502ad206a", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "5cf8e991-5cdf-408a-9a94-62c6e7c7cdb5:indexpattern-datasource-layer-47ac87f6-885e-42ca-8b55-425c2ee715a8", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "697b7157-f5cd-48e6-a896-2207f593abeb:indexpattern-datasource-layer-5c679708-1962-4585-abaf-7c09dfe3b043", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "368a354d-a2a5-4502-aba2-bc443d0fd80a:indexpattern-datasource-layer-690c8f2b-cb21-42bb-9835-4b8beabcd8b8", + "type": "index-pattern" + }, + { + "id": "sublime_security-ce2a4b74-76ca-4cdd-b3da-73530ee043c4", + "name": "e18f7396-98a9-48a7-b7cb-64416715600f:panel_e18f7396-98a9-48a7-b7cb-64416715600f", + "type": "search" + }, + { + "id": "logs-*", + "name": "controlGroup_24cd2b90-4b2c-4e94-a80a-2294e2a1a6c5:optionsListDataView", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "controlGroup_5a465a27-c8c0-441e-82b1-87e5f8cc8901:optionsListDataView", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "controlGroup_b06a8e14-3fe4-41c9-b8e2-a14e575a9def:optionsListDataView", + "type": "index-pattern" + } + ], + "type": "dashboard", + "typeMigrationVersion": "8.9.0" +} \ No newline at end of file diff --git a/packages/sublime_security/kibana/dashboard/sublime_security-f4f4e3ca-1993-4a55-9d87-a7029ee0f869.json b/packages/sublime_security/kibana/dashboard/sublime_security-f4f4e3ca-1993-4a55-9d87-a7029ee0f869.json new file mode 100644 index 00000000000..2d6482b0e65 --- /dev/null +++ b/packages/sublime_security/kibana/dashboard/sublime_security-f4f4e3ca-1993-4a55-9d87-a7029ee0f869.json @@ -0,0 +1,777 @@ +{ + "attributes": { + "description": "This dashboard shows Message Events logs collected by the Sublime Security integration.", + "kibanaSavedObjectMeta": { + "searchSourceJSON": { + "filter": [ + { + "$state": { + "store": "appState" + }, + "meta": { + "alias": null, + "disabled": false, + "field": "data_stream.dataset", + "indexRefName": "kibanaSavedObjectMeta.searchSourceJSON.filter[0].meta.index", + "key": "data_stream.dataset", + "negate": false, + "params": { + "query": "sublime_security.message_event" + }, + "type": "phrase" + }, + "query": { + "match_phrase": { + "data_stream.dataset": "sublime_security.message_event" + } + } + } + ], + "query": { + "language": "kuery", + "query": "" + } + } + }, + "optionsJSON": { + "hidePanelTitles": false, + "syncColors": false, + "syncCursor": true, + "syncTooltips": false, + "useMargins": true + }, + "panelsJSON": [ + { + "embeddableConfig": { + "enhancements": {}, + "savedVis": { + "data": { + "aggs": [], + "searchSource": { + "filter": [], + "query": { + "language": "kuery", + "query": "" + } + } + }, + "description": "", + "id": "", + "params": { + "fontSize": 12, + "markdown": "**Navigation**\n\n**Sublime Security** \n\n- [Audit](#/dashboard/sublime_security-7b4299fc-2465-46c6-bc55-dba692bb2745)\n- [Email Message](#/dashboard/sublime_security-779aade2-fbb2-425d-8647-79c2bdf2d6e0)\n- [**Message Event (This Page)**](#/dashboard/sublime_security-f4f4e3ca-1993-4a55-9d87-a7029ee0f869)\n\n**Overview**\nThis dashboard provides an overview of the message event activities from Sublime Security.\n\nThe Dashboard provides an overview of email message activities, showing the frequency of message events over time and highlighting the top 10 sender and recipient email addresses. It also displays information about spam emails and essential details of triggered actions, such as their IDs, names, and types.\n\n[**Integrations Page**](/app/integrations/detail/sublime_security-0.1.0/overview)", + "openLinksInNewTab": false + }, + "title": "", + "type": "markdown", + "uiState": {} + } + }, + "gridData": { + "h": 29, + "i": "da3fc9c4-138a-48a8-806a-f28f792d2f05", + "w": 10, + "x": 0, + "y": 0 + }, + "panelIndex": "da3fc9c4-138a-48a8-806a-f28f792d2f05", + "title": "Table of Contents", + "type": "visualization" + }, + { + "embeddableConfig": { + "attributes": { + "references": [ + { + "id": "logs-*", + "name": "indexpattern-datasource-layer-e81d7e65-4ec2-4093-a218-cc2680ba9078", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "446e564d-d953-4921-920b-0e65d9d16a4f", + "type": "index-pattern" + } + ], + "state": { + "adHocDataViews": {}, + "datasourceStates": { + "formBased": { + "layers": { + "e81d7e65-4ec2-4093-a218-cc2680ba9078": { + "columnOrder": [ + "06257ffb-165a-402f-b848-293fb29121db" + ], + "columns": { + "06257ffb-165a-402f-b848-293fb29121db": { + "customLabel": true, + "dataType": "number", + "filter": { + "language": "kuery", + "query": "" + }, + "isBucketed": false, + "label": "Spam Mails", + "operationType": "count", + "params": { + "emptyAsNull": false, + "format": { + "id": "number", + "params": { + "compact": false, + "decimals": 0 + } + } + }, + "scale": "ratio", + "sourceField": "___records___" + } + }, + "ignoreGlobalFilters": false, + "incompleteColumns": {}, + "sampling": 1 + } + } + }, + "indexpattern": { + "layers": {} + }, + "textBased": { + "layers": {} + } + }, + "filters": [ + { + "$state": { + "store": "appState" + }, + "meta": { + "alias": null, + "disabled": false, + "index": "446e564d-d953-4921-920b-0e65d9d16a4f", + "negate": false, + "params": [ + { + "meta": { + "alias": null, + "disabled": false, + "field": "data_stream.dataset", + "index": "logs-*", + "key": "data_stream.dataset", + "negate": false, + "params": { + "query": "sublime_security.message_event" + }, + "type": "phrase" + }, + "query": { + "match_phrase": { + "data_stream.dataset": "sublime_security.message_event" + } + } + }, + { + "meta": { + "alias": null, + "disabled": false, + "field": "sublime_security.message_event.landed_in_spam", + "index": "logs-*", + "key": "sublime_security.message_event.landed_in_spam", + "negate": false, + "params": { + "query": true + }, + "type": "phrase" + }, + "query": { + "match_phrase": { + "sublime_security.message_event.landed_in_spam": true + } + } + } + ], + "relation": "AND", + "type": "combined" + }, + "query": {} + } + ], + "internalReferences": [], + "query": { + "language": "kuery", + "query": "" + }, + "visualization": { + "color": "#6092C0", + "layerId": "e81d7e65-4ec2-4093-a218-cc2680ba9078", + "layerType": "data", + "metricAccessor": "06257ffb-165a-402f-b848-293fb29121db", + "showBar": false + } + }, + "title": "", + "type": "lens", + "visualizationType": "lnsMetric" + }, + "enhancements": {}, + "hidePanelTitles": true + }, + "gridData": { + "h": 13, + "i": "95ac319f-9b57-41ed-8c17-9a96d0c40155", + "w": 9, + "x": 10, + "y": 0 + }, + "panelIndex": "95ac319f-9b57-41ed-8c17-9a96d0c40155", + "title": "Spam Mails [Logs Sublime Security]", + "type": "lens" + }, + { + "embeddableConfig": { + "attributes": { + "references": [ + { + "id": "logs-*", + "name": "indexpattern-datasource-layer-344bf121-3728-45be-b07a-3f95358e8fe6", + "type": "index-pattern" + } + ], + "state": { + "adHocDataViews": {}, + "datasourceStates": { + "formBased": { + "currentIndexPatternId": "logs-*", + "layers": { + "344bf121-3728-45be-b07a-3f95358e8fe6": { + "columnOrder": [ + "45626e41-3cb0-44b6-92e4-f79d9d649ad5", + "2224ad65-7d04-41e3-b531-da37b9e40de4" + ], + "columns": { + "2224ad65-7d04-41e3-b531-da37b9e40de4": { + "customLabel": true, + "dataType": "number", + "isBucketed": false, + "label": "Message Events", + "operationType": "count", + "params": { + "emptyAsNull": false, + "format": { + "id": "number", + "params": { + "decimals": 0 + } + } + }, + "scale": "ratio", + "sourceField": "___records___" + }, + "45626e41-3cb0-44b6-92e4-f79d9d649ad5": { + "dataType": "date", + "isBucketed": true, + "label": "@timestamp", + "operationType": "date_histogram", + "params": { + "dropPartials": false, + "includeEmptyRows": true, + "interval": "auto" + }, + "scale": "interval", + "sourceField": "@timestamp" + } + }, + "ignoreGlobalFilters": false, + "incompleteColumns": {}, + "indexPatternId": "logs-*", + "sampling": 1 + } + } + }, + "indexpattern": { + "layers": {} + }, + "textBased": { + "layers": {} + } + }, + "filters": [ + { + "$state": { + "store": "appState" + }, + "meta": { + "alias": null, + "disabled": false, + "field": "data_stream.dataset", + "index": "4044d73d-b1ce-48fb-b4ff-018a1bccdeb5", + "key": "data_stream.dataset", + "negate": false, + "params": { + "query": "sublime_security.message_event" + }, + "type": "phrase" + }, + "query": { + "match_phrase": { + "data_stream.dataset": "sublime_security.message_event" + } + } + } + ], + "internalReferences": [], + "query": { + "language": "kuery", + "query": "" + }, + "visualization": { + "layers": [ + { + "accessors": [ + "2224ad65-7d04-41e3-b531-da37b9e40de4" + ], + "colorMapping": { + "assignments": [], + "colorMode": { + "type": "categorical" + }, + "paletteId": "eui_amsterdam_color_blind", + "specialAssignments": [ + { + "color": { + "type": "loop" + }, + "rule": { + "type": "other" + }, + "touched": false + } + ] + }, + "layerId": "344bf121-3728-45be-b07a-3f95358e8fe6", + "layerType": "data", + "position": "top", + "seriesType": "line", + "showGridlines": false, + "xAccessor": "45626e41-3cb0-44b6-92e4-f79d9d649ad5" + } + ], + "legend": { + "isVisible": true, + "position": "right", + "shouldTruncate": false, + "showSingleSeries": false + }, + "preferredSeriesType": "line", + "title": "Empty XY chart", + "valueLabels": "hide" + } + }, + "title": "", + "type": "lens", + "visualizationType": "lnsXY" + }, + "enhancements": {} + }, + "gridData": { + "h": 13, + "i": "cfbf8a88-05b9-4a6f-abe8-3aa43cdbda28", + "w": 29, + "x": 19, + "y": 0 + }, + "panelIndex": "cfbf8a88-05b9-4a6f-abe8-3aa43cdbda28", + "title": "Message Events over Time [Logs Sublime Security]", + "type": "lens" + }, + { + "embeddableConfig": { + "attributes": { + "references": [ + { + "id": "logs-*", + "name": "indexpattern-datasource-layer-15b1f516-2cfe-4102-8d3a-180d5e4b6b07", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "eb3e0fbd-df22-4862-90e3-b8bb93d9e4aa", + "type": "index-pattern" + } + ], + "state": { + "adHocDataViews": {}, + "datasourceStates": { + "formBased": { + "layers": { + "15b1f516-2cfe-4102-8d3a-180d5e4b6b07": { + "columnOrder": [ + "f43cd28f-2265-4013-a2f9-0ce510ce7bed", + "bd2d05e6-db1f-45c4-a57a-3876e69cef9f" + ], + "columns": { + "bd2d05e6-db1f-45c4-a57a-3876e69cef9f": { + "customLabel": true, + "dataType": "number", + "isBucketed": false, + "label": "Count", + "operationType": "count", + "params": { + "emptyAsNull": false, + "format": { + "id": "number", + "params": { + "decimals": 0 + } + } + }, + "scale": "ratio", + "sourceField": "___records___" + }, + "f43cd28f-2265-4013-a2f9-0ce510ce7bed": { + "customLabel": true, + "dataType": "string", + "isBucketed": true, + "label": "Recipient Email Address", + "operationType": "terms", + "params": { + "exclude": [], + "excludeIsRegex": false, + "include": [], + "includeIsRegex": false, + "missingBucket": false, + "orderBy": { + "columnId": "bd2d05e6-db1f-45c4-a57a-3876e69cef9f", + "type": "column" + }, + "orderDirection": "desc", + "otherBucket": false, + "parentFormat": { + "id": "terms" + }, + "secondaryFields": [], + "size": 10 + }, + "scale": "ordinal", + "sourceField": "email.to.address" + } + }, + "ignoreGlobalFilters": false, + "incompleteColumns": {}, + "sampling": 1 + } + } + }, + "indexpattern": { + "layers": {} + }, + "textBased": { + "layers": {} + } + }, + "filters": [ + { + "$state": { + "store": "appState" + }, + "meta": { + "alias": null, + "disabled": false, + "field": "data_stream.dataset", + "index": "eb3e0fbd-df22-4862-90e3-b8bb93d9e4aa", + "key": "data_stream.dataset", + "negate": false, + "params": { + "query": "sublime_security.message_event" + }, + "type": "phrase" + }, + "query": { + "match_phrase": { + "data_stream.dataset": "sublime_security.message_event" + } + } + } + ], + "internalReferences": [], + "query": { + "language": "kuery", + "query": "" + }, + "visualization": { + "columns": [ + { + "columnId": "f43cd28f-2265-4013-a2f9-0ce510ce7bed", + "isMetric": false, + "isTransposed": false + }, + { + "columnId": "bd2d05e6-db1f-45c4-a57a-3876e69cef9f", + "isMetric": true, + "isTransposed": false + } + ], + "layerId": "15b1f516-2cfe-4102-8d3a-180d5e4b6b07", + "layerType": "data" + } + }, + "title": "", + "type": "lens", + "visualizationType": "lnsDatatable" + }, + "enhancements": {} + }, + "gridData": { + "h": 16, + "i": "f21c1ce6-5aae-42be-91b7-eb5bd817e3e4", + "w": 18, + "x": 10, + "y": 13 + }, + "panelIndex": "f21c1ce6-5aae-42be-91b7-eb5bd817e3e4", + "title": "Top 10 Recipient Email [Logs Sublime Security]", + "type": "lens" + }, + { + "embeddableConfig": { + "attributes": { + "references": [ + { + "id": "logs-*", + "name": "indexpattern-datasource-layer-15b1f516-2cfe-4102-8d3a-180d5e4b6b07", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "a93be864-083b-4bcf-b5e1-9ae22d37a754", + "type": "index-pattern" + } + ], + "state": { + "adHocDataViews": {}, + "datasourceStates": { + "formBased": { + "layers": { + "15b1f516-2cfe-4102-8d3a-180d5e4b6b07": { + "columnOrder": [ + "f43cd28f-2265-4013-a2f9-0ce510ce7bed", + "bd2d05e6-db1f-45c4-a57a-3876e69cef9f" + ], + "columns": { + "bd2d05e6-db1f-45c4-a57a-3876e69cef9f": { + "customLabel": true, + "dataType": "number", + "isBucketed": false, + "label": "Count", + "operationType": "count", + "params": { + "emptyAsNull": false, + "format": { + "id": "number", + "params": { + "decimals": 0 + } + } + }, + "scale": "ratio", + "sourceField": "___records___" + }, + "f43cd28f-2265-4013-a2f9-0ce510ce7bed": { + "customLabel": true, + "dataType": "string", + "isBucketed": true, + "label": "Sender Email Address", + "operationType": "terms", + "params": { + "exclude": [], + "excludeIsRegex": false, + "include": [], + "includeIsRegex": false, + "missingBucket": false, + "orderBy": { + "columnId": "bd2d05e6-db1f-45c4-a57a-3876e69cef9f", + "type": "column" + }, + "orderDirection": "desc", + "otherBucket": false, + "parentFormat": { + "id": "terms" + }, + "size": 10 + }, + "scale": "ordinal", + "sourceField": "email.from.address" + } + }, + "ignoreGlobalFilters": false, + "incompleteColumns": {}, + "sampling": 1 + } + } + }, + "indexpattern": { + "layers": {} + }, + "textBased": { + "layers": {} + } + }, + "filters": [ + { + "$state": { + "store": "appState" + }, + "meta": { + "alias": null, + "disabled": false, + "field": "data_stream.dataset", + "index": "a93be864-083b-4bcf-b5e1-9ae22d37a754", + "key": "data_stream.dataset", + "negate": false, + "params": { + "query": "sublime_security.message_event" + }, + "type": "phrase" + }, + "query": { + "match_phrase": { + "data_stream.dataset": "sublime_security.message_event" + } + } + } + ], + "internalReferences": [], + "query": { + "language": "kuery", + "query": "" + }, + "visualization": { + "columns": [ + { + "columnId": "f43cd28f-2265-4013-a2f9-0ce510ce7bed", + "isMetric": false, + "isTransposed": false + }, + { + "columnId": "bd2d05e6-db1f-45c4-a57a-3876e69cef9f", + "isMetric": true, + "isTransposed": false + } + ], + "layerId": "15b1f516-2cfe-4102-8d3a-180d5e4b6b07", + "layerType": "data" + } + }, + "title": "", + "type": "lens", + "visualizationType": "lnsDatatable" + }, + "enhancements": {} + }, + "gridData": { + "h": 16, + "i": "3924284d-22ff-456c-88fd-7fb4041c42ef", + "w": 20, + "x": 28, + "y": 13 + }, + "panelIndex": "3924284d-22ff-456c-88fd-7fb4041c42ef", + "title": "Top 10 Sender Email [Logs Sublime Security]", + "type": "lens" + }, + { + "embeddableConfig": { + "enhancements": {}, + "rowsPerPage": 50 + }, + "gridData": { + "h": 21, + "i": "88a5d3ff-e18f-4899-96c7-56790763cc18", + "w": 48, + "x": 0, + "y": 29 + }, + "panelIndex": "88a5d3ff-e18f-4899-96c7-56790763cc18", + "panelRefName": "panel_88a5d3ff-e18f-4899-96c7-56790763cc18", + "title": "Triggered Actions Essential Details [Logs Sublime Security]", + "type": "search" + }, + { + "embeddableConfig": { + "enhancements": {} + }, + "gridData": { + "h": 21, + "i": "b0f839e4-00c2-4193-ac81-40591def58cd", + "w": 48, + "x": 0, + "y": 50 + }, + "panelIndex": "b0f839e4-00c2-4193-ac81-40591def58cd", + "panelRefName": "panel_b0f839e4-00c2-4193-ac81-40591def58cd", + "type": "search" + } + ], + "timeRestore": false, + "title": "[Logs Sublime Security] Message Event", + "version": 1 + }, + "coreMigrationVersion": "8.8.0", + "created_at": "2024-08-13T05:37:02.723Z", + "id": "sublime_security-f4f4e3ca-1993-4a55-9d87-a7029ee0f869", + "managed": false, + "references": [ + { + "id": "logs-*", + "name": "kibanaSavedObjectMeta.searchSourceJSON.filter[0].meta.index", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "95ac319f-9b57-41ed-8c17-9a96d0c40155:indexpattern-datasource-layer-e81d7e65-4ec2-4093-a218-cc2680ba9078", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "95ac319f-9b57-41ed-8c17-9a96d0c40155:446e564d-d953-4921-920b-0e65d9d16a4f", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "cfbf8a88-05b9-4a6f-abe8-3aa43cdbda28:indexpattern-datasource-layer-344bf121-3728-45be-b07a-3f95358e8fe6", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "f21c1ce6-5aae-42be-91b7-eb5bd817e3e4:indexpattern-datasource-layer-15b1f516-2cfe-4102-8d3a-180d5e4b6b07", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "f21c1ce6-5aae-42be-91b7-eb5bd817e3e4:eb3e0fbd-df22-4862-90e3-b8bb93d9e4aa", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "3924284d-22ff-456c-88fd-7fb4041c42ef:indexpattern-datasource-layer-15b1f516-2cfe-4102-8d3a-180d5e4b6b07", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "3924284d-22ff-456c-88fd-7fb4041c42ef:a93be864-083b-4bcf-b5e1-9ae22d37a754", + "type": "index-pattern" + }, + { + "id": "sublime_security-c1e2d194-7f19-46fe-bcdf-d6886edf9d3d", + "name": "88a5d3ff-e18f-4899-96c7-56790763cc18:panel_88a5d3ff-e18f-4899-96c7-56790763cc18", + "type": "search" + }, + { + "id": "sublime_security-eb590f03-79df-4189-aa74-3b5bfe20e8ca", + "name": "b0f839e4-00c2-4193-ac81-40591def58cd:panel_b0f839e4-00c2-4193-ac81-40591def58cd", + "type": "search" + } + ], + "type": "dashboard", + "typeMigrationVersion": "8.9.0" +} \ No newline at end of file diff --git a/packages/sublime_security/kibana/search/sublime_security-c1e2d194-7f19-46fe-bcdf-d6886edf9d3d.json b/packages/sublime_security/kibana/search/sublime_security-c1e2d194-7f19-46fe-bcdf-d6886edf9d3d.json new file mode 100644 index 00000000000..e44849ef38d --- /dev/null +++ b/packages/sublime_security/kibana/search/sublime_security-c1e2d194-7f19-46fe-bcdf-d6886edf9d3d.json @@ -0,0 +1,83 @@ +{ + "attributes": { + "columns": [ + "sublime_security.message_event.data.triggered_actions.id", + "sublime_security.message_event.data.triggered_actions.name", + "sublime_security.message_event.data.triggered_actions.type" + ], + "description": "", + "grid": {}, + "hideChart": false, + "isTextBasedQuery": false, + "kibanaSavedObjectMeta": { + "searchSourceJSON": { + "filter": [ + { + "$state": { + "store": "appState" + }, + "meta": { + "alias": null, + "disabled": false, + "field": "data_stream.dataset", + "indexRefName": "kibanaSavedObjectMeta.searchSourceJSON.filter[0].meta.index", + "key": "data_stream.dataset", + "negate": false, + "params": { + "query": "sublime_security.message_event" + }, + "type": "phrase" + }, + "query": { + "match_phrase": { + "data_stream.dataset": "sublime_security.message_event" + } + } + } + ], + "indexRefName": "kibanaSavedObjectMeta.searchSourceJSON.index", + "query": { + "language": "kuery", + "query": "" + } + } + }, + "sort": [ + [ + "@timestamp", + "desc" + ] + ], + "timeRestore": false, + "title": "Triggered Action Essential Details [Logs Sublime Security]", + "usesAdHocDataView": false + }, + "coreMigrationVersion": "8.8.0", + "created_at": "2024-08-13T05:21:24.585Z", + "id": "sublime_security-c1e2d194-7f19-46fe-bcdf-d6886edf9d3d", + "managed": true, + "references": [ + { + "id": "logs-*", + "name": "kibanaSavedObjectMeta.searchSourceJSON.index", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "kibanaSavedObjectMeta.searchSourceJSON.filter[0].meta.index", + "type": "index-pattern" + }, + { + "id": "sublime_security-security-solution-default", + "name": "tag-ref-sublime_security-security-solution-default", + "type": "tag" + }, + { + "id": "sublime_security-security-solution-default", + "name": "tag-ref-security-solution-default", + "type": "tag" + } + ], + "type": "search", + "typeMigrationVersion": "10.2.0" +} \ No newline at end of file diff --git a/packages/sublime_security/kibana/search/sublime_security-ce2a4b74-76ca-4cdd-b3da-73530ee043c4.json b/packages/sublime_security/kibana/search/sublime_security-ce2a4b74-76ca-4cdd-b3da-73530ee043c4.json new file mode 100644 index 00000000000..d9a59c62e0e --- /dev/null +++ b/packages/sublime_security/kibana/search/sublime_security-ce2a4b74-76ca-4cdd-b3da-73530ee043c4.json @@ -0,0 +1,101 @@ +{ + "attributes": { + "columns": [ + "source.ip", + "http.request.id", + "http.request.method", + "url.path", + "user.email", + "user.roles" + ], + "description": "", + "grid": { + "columns": { + "@timestamp": { + "width": 217 + }, + "http.request.id": { + "width": 305 + }, + "http.request.method": { + "width": 138 + }, + "source.ip": { + "width": 145 + } + } + }, + "hideChart": false, + "isTextBasedQuery": false, + "kibanaSavedObjectMeta": { + "searchSourceJSON": { + "filter": [ + { + "$state": { + "store": "appState" + }, + "meta": { + "alias": null, + "disabled": false, + "field": "data_stream.dataset", + "indexRefName": "kibanaSavedObjectMeta.searchSourceJSON.filter[0].meta.index", + "key": "data_stream.dataset", + "negate": false, + "params": { + "query": "sublime_security.audit" + }, + "type": "phrase" + }, + "query": { + "match_phrase": { + "data_stream.dataset": "sublime_security.audit" + } + } + } + ], + "indexRefName": "kibanaSavedObjectMeta.searchSourceJSON.index", + "query": { + "language": "kuery", + "query": "" + } + } + }, + "sort": [ + [ + "@timestamp", + "desc" + ] + ], + "timeRestore": false, + "title": "Audit Event Essential Details", + "usesAdHocDataView": false + }, + "coreMigrationVersion": "8.8.0", + "created_at": "2024-08-13T05:21:24.585Z", + "id": "sublime_security-ce2a4b74-76ca-4cdd-b3da-73530ee043c4", + "managed": true, + "references": [ + { + "id": "logs-*", + "name": "kibanaSavedObjectMeta.searchSourceJSON.index", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "kibanaSavedObjectMeta.searchSourceJSON.filter[0].meta.index", + "type": "index-pattern" + }, + { + "id": "sublime_security-security-solution-default", + "name": "tag-ref-sublime_security-security-solution-default", + "type": "tag" + }, + { + "id": "sublime_security-security-solution-default", + "name": "tag-ref-security-solution-default", + "type": "tag" + } + ], + "type": "search", + "typeMigrationVersion": "10.2.0" +} \ No newline at end of file diff --git a/packages/sublime_security/kibana/search/sublime_security-eb590f03-79df-4189-aa74-3b5bfe20e8ca.json b/packages/sublime_security/kibana/search/sublime_security-eb590f03-79df-4189-aa74-3b5bfe20e8ca.json new file mode 100644 index 00000000000..5f648e345f2 --- /dev/null +++ b/packages/sublime_security/kibana/search/sublime_security-eb590f03-79df-4189-aa74-3b5bfe20e8ca.json @@ -0,0 +1,84 @@ +{ + "attributes": { + "columns": [ + "sublime_security.message_event.mailbox.id", + "sublime_security.message_event.mailbox.email", + "sublime_security.message_event.mailbox.external_id", + "sublime_security.message_event.landed_in_spam" + ], + "description": "", + "grid": {}, + "hideChart": false, + "isTextBasedQuery": false, + "kibanaSavedObjectMeta": { + "searchSourceJSON": { + "filter": [ + { + "$state": { + "store": "appState" + }, + "meta": { + "alias": null, + "disabled": false, + "field": "data_stream.dataset", + "indexRefName": "kibanaSavedObjectMeta.searchSourceJSON.filter[0].meta.index", + "key": "data_stream.dataset", + "negate": false, + "params": { + "query": "sublime_security.message_event" + }, + "type": "phrase" + }, + "query": { + "match_phrase": { + "data_stream.dataset": "sublime_security.message_event" + } + } + } + ], + "indexRefName": "kibanaSavedObjectMeta.searchSourceJSON.index", + "query": { + "language": "kuery", + "query": "" + } + } + }, + "sort": [ + [ + "@timestamp", + "desc" + ] + ], + "timeRestore": false, + "title": "Mailbox Essential Details [Logs Sublime Security]", + "usesAdHocDataView": false + }, + "coreMigrationVersion": "8.8.0", + "created_at": "2024-08-13T05:21:24.585Z", + "id": "sublime_security-eb590f03-79df-4189-aa74-3b5bfe20e8ca", + "managed": true, + "references": [ + { + "id": "logs-*", + "name": "kibanaSavedObjectMeta.searchSourceJSON.index", + "type": "index-pattern" + }, + { + "id": "logs-*", + "name": "kibanaSavedObjectMeta.searchSourceJSON.filter[0].meta.index", + "type": "index-pattern" + }, + { + "id": "sublime_security-security-solution-default", + "name": "tag-ref-sublime_security-security-solution-default", + "type": "tag" + }, + { + "id": "sublime_security-security-solution-default", + "name": "tag-ref-security-solution-default", + "type": "tag" + } + ], + "type": "search", + "typeMigrationVersion": "10.2.0" +} \ No newline at end of file diff --git a/packages/sublime_security/kibana/tag/sublime_security-security-solution-default.json b/packages/sublime_security/kibana/tag/sublime_security-security-solution-default.json new file mode 100644 index 00000000000..a21917e2100 --- /dev/null +++ b/packages/sublime_security/kibana/tag/sublime_security-security-solution-default.json @@ -0,0 +1,14 @@ +{ + "attributes": { + "color": "#00BFB3", + "description": "Tag defined in package-spec", + "name": "Security Solution" + }, + "coreMigrationVersion": "8.8.0", + "created_at": "2024-08-13T05:21:24.630Z", + "id": "sublime_security-security-solution-default", + "managed": false, + "references": [], + "type": "tag", + "typeMigrationVersion": "8.0.0" +} \ No newline at end of file diff --git a/packages/sublime_security/kibana/tags.yml b/packages/sublime_security/kibana/tags.yml new file mode 100644 index 00000000000..47f20a8f551 --- /dev/null +++ b/packages/sublime_security/kibana/tags.yml @@ -0,0 +1,4 @@ +- text: Security Solution + asset_types: + - dashboard + - search diff --git a/packages/sublime_security/manifest.yml b/packages/sublime_security/manifest.yml new file mode 100644 index 00000000000..70e364f63b4 --- /dev/null +++ b/packages/sublime_security/manifest.yml @@ -0,0 +1,225 @@ +format_version: 3.2.1 +name: sublime_security +title: Sublime Security +version: 0.1.0 +description: Collect logs from Sublime Security with Elastic Agent. +type: integration +categories: + - security + - email_security +conditions: + kibana: + version: '^8.13.0' + elastic: + subscription: basic +screenshots: + - src: /img/sublime_security-audit.png + title: sublime security audit dashboard + size: 600x600 + type: image/png + - src: /img/sublime_security-email_message.png + title: sublime security email message dashboard + size: 600x600 + type: image/png + - src: /img/sublime_security-message_event.png + title: sublime security message event dashboard + size: 600x600 + type: image/png +icons: + - src: /img/sublime_security.svg + title: Sublime Security logo + size: 32x32 + type: image/svg+xml + - src: /img/sublime_security-dark.svg + title: Sublime Security dark logo + size: 32x32 + type: image/svg+xml + dark_mode: true +policy_templates: + - name: sublime_security + title: Sublime Security logs + description: Collect Sublime Security logs. + inputs: + - type: cel + title: Collect Sublime Security logs via API + description: Collecting Sublime Security logs via API. + vars: + - name: url + type: url + title: URL + default: https://api.platform.sublimesecurity.com + description: Base URL of the Sublime Security API. Depending on type of deployment, URL may differ. + multi: false + required: true + show_user: false + - name: api_key + type: password + title: API Key + description: API Key of the Sublime Security API. + multi: false + required: true + show_user: true + secret: true + - name: proxy_url + type: text + title: Proxy URL + multi: false + required: false + show_user: false + description: URL to proxy connections in the form of http[s]://:@:. Please ensure your username and password are in URL encoded format. + - name: ssl + type: yaml + title: SSL Configuration + description: i.e. certificate_authorities, supported_protocols, verification_mode etc. + multi: false + required: false + show_user: false + default: | + #certificate_authorities: + # - | + # -----BEGIN CERTIFICATE----- + # MIIDCjCCAfKgAwIBAgITJ706Mu2wJlKckpIvkWxEHvEyijANBgkqhkiG9w0BAQsF + # ADAUMRIwEAYDVQQDDAlsb2NhbGhvc3QwIBcNMTkwNzIyMTkyOTA0WhgPMjExOTA2 + # MjgxOTI5MDRaMBQxEjAQBgNVBAMMCWxvY2FsaG9zdDCCASIwDQYJKoZIhvcNAQEB + # BQADggEPADCCAQoCggEBANce58Y/JykI58iyOXpxGfw0/gMvF0hUQAcUrSMxEO6n + # fZRA49b4OV4SwWmA3395uL2eB2NB8y8qdQ9muXUdPBWE4l9rMZ6gmfu90N5B5uEl + # 94NcfBfYOKi1fJQ9i7WKhTjlRkMCgBkWPkUokvBZFRt8RtF7zI77BSEorHGQCk9t + # /D7BS0GJyfVEhftbWcFEAG3VRcoMhF7kUzYwp+qESoriFRYLeDWv68ZOvG7eoWnP + # PsvZStEVEimjvK5NSESEQa9xWyJOmlOKXhkdymtcUd/nXnx6UTCFgnkgzSdTWV41 + # CI6B6aJ9svCTI2QuoIq2HxX/ix7OvW1huVmcyHVxyUECAwEAAaNTMFEwHQYDVR0O + # BBYEFPwN1OceFGm9v6ux8G+DZ3TUDYxqMB8GA1UdIwQYMBaAFPwN1OceFGm9v6ux + # 8G+DZ3TUDYxqMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBAG5D + # 874A4YI7YUwOVsVAdbWtgp1d0zKcPRR+r2OdSbTAV5/gcS3jgBJ3i1BN34JuDVFw + # 3DeJSYT3nxy2Y56lLnxDeF8CUTUtVQx3CuGkRg1ouGAHpO/6OqOhwLLorEmxi7tA + # H2O8mtT0poX5AnOAhzVy7QW0D/k4WaoLyckM5hUa6RtvgvLxOwA0U+VGurCDoctu + # 8F4QOgTAWyh8EZIwaKCliFRSynDpv3JTUwtfZkxo6K6nce1RhCWFAsMvDZL8Dgc0 + # yvgJ38BRsFOtkRuAGSf6ZUwTO8JJRRIFnpUzXflAnGivK9M13D5GEQMmIl6U9Pvk + # sxSmbIUfc2SGJGCJD4I= + # -----END CERTIFICATE----- + - type: aws-s3 + title: Collect Sublime Security logs via AWS S3 or AWS SQS + description: Collecting logs from Sublime Security via AWS S3 or AWS SQS. + vars: + - name: collect_s3_logs + required: true + show_user: true + title: Collect logs via S3 Bucket + description: To collect logs via S3 bucket enable the toggle switch. By default, it will collect logs via SQS Queue. + type: bool + multi: false + default: false + - name: access_key_id + type: password + title: Access Key ID + multi: false + required: false + show_user: true + description: First part of access key. This parameter along with the secret_access_key parameter is required if we are not providing shared_credential_file. + secret: true + - name: secret_access_key + type: password + title: Secret Access Key + multi: false + required: false + show_user: true + description: Second part of access key. This parameter along with the access_key_id parameter is required if we are not providing shared_credential_file. + secret: true + - name: region + type: text + title: '[SQS] Region' + multi: false + required: false + show_user: true + description: The name of the AWS region of the end point. If this option is given it takes precedence over the region name obtained from the queue_url value. + - name: session_token + type: password + title: Session Token + multi: false + required: false + show_user: true + description: Required when using temporary security credentials. + secret: true + - name: shared_credential_file + type: text + title: Shared Credential File + multi: false + required: false + show_user: false + description: Directory of the shared credentials file. This parameter is required if we are not providing value for the parameters - secret_access_key and access_key_id. + - name: credential_profile_name + type: text + title: Credential Profile Name + multi: false + required: false + show_user: false + description: Profile name in shared credentials file. + - name: role_arn + type: text + title: Role ARN + multi: false + required: false + show_user: false + description: AWS IAM Role to assume. + - name: default_region + type: text + title: Default AWS Region + multi: false + required: false + show_user: false + default: "" + description: >- + Default region to query if no other region is set. Most AWS services offer a regional endpoint that can be used to make requests. Some services, such as IAM, do not support regions. If a region is not provided by any other way (environment variable, credential or instance profile), the value set here will be used. + - name: endpoint + type: text + title: Endpoint + multi: false + required: false + show_user: false + description: URL of the entry point for an AWS web service. + - name: fips_enabled + type: bool + title: FIPS Enabled + default: false + multi: false + required: false + show_user: false + description: Enabling this option changes the service name from `s3` to `s3-fips` for connecting to the correct service endpoint. + - name: proxy_url + type: text + title: Proxy URL + multi: false + required: false + show_user: false + description: URL to proxy connections in the form of http[s]://:@:. Please ensure your username and password are in URL encoded format. + - name: ssl + type: yaml + title: SSL Configuration + description: i.e. certificate_authorities, supported_protocols, verification_mode etc. + multi: false + required: false + show_user: false + default: | + #certificate_authorities: + # - | + # -----BEGIN CERTIFICATE----- + # MIIDCjCCAfKgAwIBAgITJ706Mu2wJlKckpIvkWxEHvEyijANBgkqhkiG9w0BAQsF + # ADAUMRIwEAYDVQQDDAlsb2NhbGhvc3QwIBcNMTkwNzIyMTkyOTA0WhgPMjExOTA2 + # MjgxOTI5MDRaMBQxEjAQBgNVBAMMCWxvY2FsaG9zdDCCASIwDQYJKoZIhvcNAQEB + # BQADggEPADCCAQoCggEBANce58Y/JykI58iyOXpxGfw0/gMvF0hUQAcUrSMxEO6n + # fZRA49b4OV4SwWmA3395uL2eB2NB8y8qdQ9muXUdPBWE4l9rMZ6gmfu90N5B5uEl + # 94NcfBfYOKi1fJQ9i7WKhTjlRkMCgBkWPkUokvBZFRt8RtF7zI77BSEorHGQCk9t + # /D7BS0GJyfVEhftbWcFEAG3VRcoMhF7kUzYwp+qESoriFRYLeDWv68ZOvG7eoWnP + # PsvZStEVEimjvK5NSESEQa9xWyJOmlOKXhkdymtcUd/nXnx6UTCFgnkgzSdTWV41 + # CI6B6aJ9svCTI2QuoIq2HxX/ix7OvW1huVmcyHVxyUECAwEAAaNTMFEwHQYDVR0O + # BBYEFPwN1OceFGm9v6ux8G+DZ3TUDYxqMB8GA1UdIwQYMBaAFPwN1OceFGm9v6ux + # 8G+DZ3TUDYxqMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBAG5D + # 874A4YI7YUwOVsVAdbWtgp1d0zKcPRR+r2OdSbTAV5/gcS3jgBJ3i1BN34JuDVFw + # 3DeJSYT3nxy2Y56lLnxDeF8CUTUtVQx3CuGkRg1ouGAHpO/6OqOhwLLorEmxi7tA + # H2O8mtT0poX5AnOAhzVy7QW0D/k4WaoLyckM5hUa6RtvgvLxOwA0U+VGurCDoctu + # 8F4QOgTAWyh8EZIwaKCliFRSynDpv3JTUwtfZkxo6K6nce1RhCWFAsMvDZL8Dgc0 + # yvgJ38BRsFOtkRuAGSf6ZUwTO8JJRRIFnpUzXflAnGivK9M13D5GEQMmIl6U9Pvk + # sxSmbIUfc2SGJGCJD4I= + # -----END CERTIFICATE----- +owner: + github: elastic/security-service-integrations + type: elastic diff --git a/packages/sublime_security/validation.yml b/packages/sublime_security/validation.yml new file mode 100644 index 00000000000..1189aa63c89 --- /dev/null +++ b/packages/sublime_security/validation.yml @@ -0,0 +1,3 @@ +errors: + exclude_checks: + - SVR00004 # References in dashboards.