[Discuss] OpenSearch Dashboards Engine Plugin

[kavilla]

The following issues:

• Allow DQL search or publish parsing library to generate query from DQL  OpenSearch#9498
• Verify Usage of Opensearch Core System and Hidden Indices #4713

Kind of revolve the idea that there are things within OpenSearch Dashboards that can be placed within OpenSearch.

#4713 comments on the usage of "system" index of OpenSearch Dashboards to which we store our saved objects in (dashboards, visualizations, filters, etc). We mimic the prefix . to indicate that it's a system index but since there is no plugin on the engine side specifically related to OpenSearch Dashboards it is not actually a system index. Even though it is a system index for us, it is not for OpenSearch and breaks from standard. To actually make this a system index within the environment, OpenSearch Dashboards would need to create a plugin within OpenSearch and extend the system index plugin.

opensearch-project/OpenSearch#9498 provides an opportunity that users would like to utilize the simplicity of the DQL APIs to query OpenSearch without querying OpenSearch Dashboards. Which also gives way to potentially placing the DQL APIs within engine plugin for OpenSearch Dashboards that contains specific APIs.

Can be extended to other APIs as well. However, it comes at the cost where we are managing more items per release. If we place it within core then we would be handing off responsibility which is not a positive due to potential breaking changes.

Question

Should we consider moving some of the stuff from OpenSearch Dashboards to OpenSearch?

[joshuarrrr]

One of the key considerations is understanding which minimal/bundled configurations are most commonly useful for people. For example, the relative demand for:

1. OpenSearch (minimal) - some use cases don't require OpenSearch Dashboards at all
2. OpenSearch (with plugins)
3. OpenSearch + OpenSearch Dashboards (minimal)
4. OpenSearch + OpenSearch Dashboards (with plugins)

Anecdotally, we see users very much choosing 4, and it's worth considering how real the differing demand for 1 and 3 are.

I'd also be curious what @zengyan-amazon or @bandinib-amzn think about this idea, given that some of their work involves further separating OpenSearch Dashboards from its dependency on OpenSearch.