Skip to content

Commit

Permalink
Merge branch 'main' into update-data-sources-S3
Browse files Browse the repository at this point in the history
  • Loading branch information
vagimeli authored Aug 29, 2024
2 parents 1fb3d37 + 2b2b971 commit de1cf2e
Show file tree
Hide file tree
Showing 29 changed files with 1,260 additions and 385 deletions.
1 change: 1 addition & 0 deletions .github/vale/styles/Vocab/OpenSearch/Words/accept.txt
Original file line number Diff line number Diff line change
Expand Up @@ -127,6 +127,7 @@ stdout
[Ss]ubvector
[Ss]ubwords?
[Ss]uperset
[Ss]uperadmins?
[Ss]yslog
tebibyte
[Tt]emplated
Expand Down
52 changes: 30 additions & 22 deletions _about/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,16 +22,21 @@ This section contains documentation for OpenSearch and OpenSearch Dashboards.

## Getting started

- [Intro to OpenSearch]({{site.url}}{{site.baseurl}}/intro/)
- [Quickstart]({{site.url}}{{site.baseurl}}/quickstart/)
To get started, explore the following documentation:

- [Getting started guide]({{site.url}}{{site.baseurl}}/getting-started/):
- [Intro to OpenSearch]({{site.url}}{{site.baseurl}}/getting-started/intro/)
- [Installation quickstart]({{site.url}}{{site.baseurl}}/getting-started/quickstart/)
- [Communicate with OpenSearch]({{site.url}}{{site.baseurl}}/getting-started/communicate/)
- [Ingest data]({{site.url}}{{site.baseurl}}/getting-started/ingest-data/)
- [Search data]({{site.url}}{{site.baseurl}}/getting-started/search-data/)
- [Getting started with OpenSearch security]({{site.url}}{{site.baseurl}}/getting-started/security/)
- [Install OpenSearch]({{site.url}}{{site.baseurl}}/install-and-configure/install-opensearch/index/)
- [Install OpenSearch Dashboards]({{site.url}}{{site.baseurl}}/install-and-configure/install-dashboards/index/)
- [See the FAQ](https://opensearch.org/faq)
- [FAQ](https://opensearch.org/faq)

## Why use OpenSearch?

With OpenSearch, you can perform the following use cases:

<table style="table-layout: auto ; width: 100%;">
<tbody>
<tr style="text-align: center; vertical-align:center;">
Expand All @@ -41,35 +46,38 @@ With OpenSearch, you can perform the following use cases:
<td><img src="{{site.url}}{{site.baseurl}}/images/4_tracking.png" class="no-border" alt="Operational health tracking" height="100"/></td>
</tr>
<tr style="text-align: left; vertical-align:top; font-weight: bold; color: rgb(0,59,92)">
<td>Fast, Scalable Full-text Search</td>
<td>Application and Infrastructure Monitoring</td>
<td>Security and Event Information Management</td>
<td>Operational Health Tracking</td>
<td>Fast, scalable full-text search</td>
<td>Application and infrastructure monitoring</td>
<td>Security and event information management</td>
<td>Operational health tracking</td>
</tr>
<tr style="text-align: left; vertical-align:top;">
<td>Help users find the right information within your application, website, or data lake catalog. </td>
<td>Easily store and analyze log data, and set automated alerts for underperformance.</td>
<td>Easily store and analyze log data, and set automated alerts for performance issues.</td>
<td>Centralize logs to enable real-time security monitoring and forensic analysis.</td>
<td>Use observability logs, metrics, and traces to monitor your applications and business in real time.</td>
<td>Use observability logs, metrics, and traces to monitor your applications in real time.</td>
</tr>
</tbody>
</table>

**Additional features and plugins:**
## Key features

OpenSearch provides several features to help index, secure, monitor, and analyze your data:

OpenSearch has several features and plugins to help index, secure, monitor, and analyze your data. Most OpenSearch plugins have corresponding OpenSearch Dashboards plugins that provide a convenient, unified user interface.
- [Anomaly detection]({{site.url}}{{site.baseurl}}/monitoring-plugins/ad/) - Identify atypical data and receive automatic notifications
- [KNN]({{site.url}}{{site.baseurl}}/search-plugins/knn/) - Find “nearest neighbors” in your vector data
- [Performance Analyzer]({{site.url}}{{site.baseurl}}/monitoring-plugins/pa/) - Monitor and optimize your cluster
- [SQL]({{site.url}}{{site.baseurl}}/search-plugins/sql/index/) - Use SQL or a piped processing language to query your data
- [Index State Management]({{site.url}}{{site.baseurl}}/im-plugin/) - Automate index operations
- [ML Commons plugin]({{site.url}}{{site.baseurl}}/ml-commons-plugin/index/) - Train and execute machine-learning models
- [Asynchronous search]({{site.url}}{{site.baseurl}}/search-plugins/async/) - Run search requests in the background
- [Cross-cluster replication]({{site.url}}{{site.baseurl}}/replication-plugin/index/) - Replicate your data across multiple OpenSearch clusters
- [Anomaly detection]({{site.url}}{{site.baseurl}}/monitoring-plugins/ad/) -- Identify atypical data and receive automatic notifications.
- [SQL]({{site.url}}{{site.baseurl}}/search-plugins/sql/index/) -- Use SQL or a Piped Processing Language (PPL) to query your data.
- [Index State Management]({{site.url}}{{site.baseurl}}/im-plugin/) -- Automate index operations.
- [Search methods]({{site.url}}{{site.baseurl}}/search-plugins/knn/) -- From traditional lexical search to advanced vector and hybrid search, discover the optimal search method for your use case.
- [Machine learning]({{site.url}}{{site.baseurl}}/ml-commons-plugin/index/) -- Integrate machine learning models into your workloads.
- [Workflow automation]({{site.url}}{{site.baseurl}}/automating-configurations/index/) -- Automate complex OpenSearch setup and preprocessing tasks.
- [Performance evaluation]({{site.url}}{{site.baseurl}}/monitoring-plugins/pa/) -- Monitor and optimize your cluster.
- [Asynchronous search]({{site.url}}{{site.baseurl}}/search-plugins/async/) -- Run search requests in the background.
- [Cross-cluster replication]({{site.url}}{{site.baseurl}}/replication-plugin/index/) -- Replicate your data across multiple OpenSearch clusters.


## The secure path forward
OpenSearch includes a demo configuration so that you can get up and running quickly, but before using OpenSearch in a production environment, you must [configure the Security plugin manually]({{site.url}}{{site.baseurl}}/security/configuration/index/) with your own certificates, authentication method, users, and passwords.

OpenSearch includes a demo configuration so that you can get up and running quickly, but before using OpenSearch in a production environment, you must [configure the Security plugin manually]({{site.url}}{{site.baseurl}}/security/configuration/index/) with your own certificates, authentication method, users, and passwords. To get started, see [Getting started with OpenSearch security]({{site.url}}{{site.baseurl}}/getting-started/security/).

## Looking for the Javadoc?

Expand Down
4 changes: 3 additions & 1 deletion _automating-configurations/api/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,4 +18,6 @@ OpenSearch supports the following workflow APIs:
* [Search workflow]({{site.url}}{{site.baseurl}}/automating-configurations/api/search-workflow/)
* [Search workflow state]({{site.url}}{{site.baseurl}}/automating-configurations/api/search-workflow-state/)
* [Deprovision workflow]({{site.url}}{{site.baseurl}}/automating-configurations/api/deprovision-workflow/)
* [Delete workflow]({{site.url}}{{site.baseurl}}/automating-configurations/api/delete-workflow/)
* [Delete workflow]({{site.url}}{{site.baseurl}}/automating-configurations/api/delete-workflow/)

For information about workflow access control, see [Workflow template security]({{site.url}}{{site.baseurl}}/automating-configurations/workflow-security/).
1 change: 1 addition & 0 deletions _automating-configurations/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,3 +44,4 @@ Workflow automation provides the following benefits:
- For the workflow step syntax, see [Workflow steps]({{site.url}}{{site.baseurl}}/automating-configurations/workflow-steps/).
- For a complete example, see [Workflow tutorial]({{site.url}}{{site.baseurl}}/automating-configurations/workflow-tutorial/).
- For configurable settings, see [Workflow settings]({{site.url}}{{site.baseurl}}/automating-configurations/workflow-settings/).
- For information about workflow access control, see [Workflow template security]({{site.url}}{{site.baseurl}}/automating-configurations/workflow-security/).
93 changes: 93 additions & 0 deletions _automating-configurations/workflow-security.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
---
layout: default
title: Workflow template security
nav_order: 50
---

# Workflow template security

In OpenSearch, automated workflow configurations are provided by the Flow Framework plugin. You can use the Security plugin together with the Flow Framework plugin to limit non-admin users to specific actions. For example, you might want some users to only be able to create, update, or delete workflows, while others may only be able to view workflows.

All Flow Framework indexes are protected as system indexes. Only a superadmin user or an admin user with a TLS certificate can access system indexes. For more information, see [System indexes]({{site.url}}{{site.baseurl}}/security/configuration/system-indices/).

Security for Flow Framework is set up similarly to [security for anomaly detection]({{site.url}}{{site.baseurl}}/monitoring-plugins/ad/security/).

## Basic permissions

As an admin user, you can use the Security plugin to assign specific permissions to users based on the APIs they need to access. For a list of supported Flow Framework APIs, see [Workflow APIs]({{site.url}}{{site.baseurl}}/automating-configurations/api/index/).

The Security plugin has two built-in roles that cover most Flow Framework use cases: `flow_framework_full_access` and `flow_framework_read_access`. For descriptions of each, see [Predefined roles]({{site.url}}{{site.baseurl}}/security/access-control/users-roles#predefined-roles).

If these roles don't meet your needs, you can assign users individual Flow Framework [permissions]({{site.url}}{{site.baseurl}}/security/access-control/permissions/) to suit your use case. Each action corresponds to an operation in the REST API. For example, the `cluster:admin/opensearch/flow_framework/workflow/search` permission lets you search workflows.

### Fine-grained access control

To reduce the chances of unintended users viewing metadata that describes an index, we recommend that administrators enable role-based access control when assigning permissions to the intended user group. For more information, see [Limit access by backend role](#advanced-limit-access-by-backend-role).

## (Advanced) Limit access by backend role

Use backend roles to configure fine-grained access to individual workflows based on roles. For example, users in different departments of an organization can view workflows owned by their own department.

First, make sure your users have the appropriate [backend roles]({{site.url}}{{site.baseurl}}/security/access-control/index/). Backend roles usually come from an [LDAP server]({{site.url}}{{site.baseurl}}/security/configuration/ldap/) or [SAML provider]({{site.url}}{{site.baseurl}}/security/configuration/saml/), but if you use an internal user database, you can [create users manually using the API]({{site.url}}{{site.baseurl}}/security/access-control/api#create-user).

Next, enable the following setting:

```json
PUT _cluster/settings
{
"transient": {
"plugins.flow_framework.filter_by_backend_roles": "true"
}
}
```
{% include copy-curl.html %}

Now when users view workflow resources in OpenSearch Dashboards (or make REST API calls), they only see workflows created by users who share at least one backend role.

For example, consider two users: `alice` and `bob`.

`alice` has an `analyst` backend role:

```json
PUT _plugins/_security/api/internalusers/alice
{
"password": "alice",
"backend_roles": [
"analyst"
],
"attributes": {}
}
```

`bob` has a `human-resources` backend role:

```json
PUT _plugins/_security/api/internalusers/bob
{
"password": "bob",
"backend_roles": [
"human-resources"
],
"attributes": {}
}
```

Both `alice` and `bob` have full access to the Flow Framework APIs:

```json
PUT _plugins/_security/api/rolesmapping/flow_framework_full_access
{
"backend_roles": [],
"hosts": [],
"users": [
"alice",
"bob"
]
}
```

Because they have different backend roles, `alice` and `bob` cannot view each other's workflows or their results.

Users without backend roles can still view other users' workflow results if they have `flow_framework_read_access`. This also applies to users who have `flow_framework_full_access` because this permission includes all of the permissions of `flow_framework_read_access`.

Administrators should inform users that the `flow_framework_read_access` permission allows them to view the results of any workflow in a cluster, including data not directly accessible to them. To limit access to the results of a specific workflow, administrators should apply backend role filters when creating the workflow. This ensures that only users with matching backend roles can access that workflow's results.
53 changes: 53 additions & 0 deletions _dashboards/management/connect-prometheus.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
---
layout: default
title: Connecting Prometheus to OpenSearch
parent: Data sources
nav_order: 20
---

# Connecting Prometheus to OpenSearch
Introduced 2.16
{: .label .label-purple }

This documentation covers the key steps to connect Prometheus to OpenSearch using the OpenSearch Dashboards interface, including setting up the data source connection, modifying the connection details, and creating an index pattern for the Prometheus data.

## Prerequisites and permissions

Before connecting a data source, ensure you have met the [Prerequisites]({{site.url}}{{site.baseurl}}/dashboards/management/data-sources/#prerequisites) and have the necessary [Permissions]({{site.url}}{{site.baseurl}}/dashboards/management/data-sources/#permissions).

## Create a Prometheus data source connection

A data source connection specifies the parameters needed to connect to a data source. These parameters form a connection string for the data source. Using OpenSearch Dashboards, you can add new **Prometheus** data source connections or manage existing ones.

Follow these steps to connect your data source:

1. From the OpenSearch Dashboards main menu, go to **Management** > **Data sources** > **New data source** > **Prometheus**.

2. From the **Configure Prometheus data source** section:

- Under **Data source details**, provide a title and optional description.
- Under **Prometheus data location**, enter the Prometheus URI.
- Under **Authentication details**, select the appropriate authentication method from the dropdown list and enter the required details:
- **Basic authentication**: Enter a username and password.
- **AWS Signature Version 4**: Specify the **Region**, select the OpenSearch service from the **Service Name** list (**Amazon OpenSearch Service** or **Amazon OpenSearch Serverless**), and enter the **Access Key** and **Secret Key**.
- Under **Query permissions**, choose the role needed to search and index data. If you select **Restricted**, an additional field will become available to configure the required role.

3. Select **Review Configuration** > **Connect to Prometheus** to save your settings. The new connection will appear in the list of data sources.

## Modify a data source connection

To modify a data source connection, follow these steps:

1. Select the desired connection from the list on the **Data sources** main page. This will open the **Connection Details** window.
2. Within the **Connection Details** window, edit the **Title** and **Description** fields. Select the **Save changes** button to apply the changes.
3. To update the **Authentication Method**, choose the method from the dropdown list and enter any necessary credentials. Select **Save changes** to apply the changes.
- To update the **Basic authentication** authentication method, select the **Update stored password** button. Within the pop-up window, enter the updated password and confirm it and select **Update stored password** to save the changes. To test the connection, select the **Test connection** button.
- To update the **AWS Signature Version 4** authentication method, select the **Update stored AWS credential** button. Within the pop-up window, enter the updated access and secret keys and select **Update stored AWS credential** to save the changes. To test the connection, select the **Test connection** button.

## Delete a data source connection

To delete the data source connection, select the {::nomarkdown}<img src="{{site.url}}{{site.baseurl}}/images/dashboards/trash-can-icon.png" class="inline-icon" alt="delete icon"/>{:/} icon.

## Create an index pattern

After creating a data source connection, the next step is to create an index pattern for that data source. For more information and a tutorial on index patterns, refer to [Index patterns]({{site.url}}{{site.baseurl}}/dashboards/management/index-patterns/).
Loading

0 comments on commit de1cf2e

Please sign in to comment.