diff --git a/README.md b/README.md index 181a3a4d..f377ea2a 100644 --- a/README.md +++ b/README.md @@ -102,6 +102,10 @@ See the [design doc](https://github.com/opensearch-project/performance-analyzer- Please refer to the [technical documentation](https://opensearch.org/docs/monitoring-plugins/pa/index/) for detailed information on installing and configuring Performance Analyzer. +The [Architecture](docs/READER.md) document provides a high-level overview of the Performance Analyzer architecture. + +You can find a high-level overview of the project file structure [here](docs/OVERVIEW.md). + ## Contributing See [developer guide](DEVELOPER_GUIDE.md) and [how to contribute to this project](CONTRIBUTING.md). diff --git a/docs/OVERVIEW.md b/docs/OVERVIEW.md new file mode 100644 index 00000000..98a89b4f --- /dev/null +++ b/docs/OVERVIEW.md @@ -0,0 +1,30 @@ +In `src/main/java/org/opensearch/performanceanalyzer/`: + +- `PerformanceAnalyzerPlugin`'s constructor does the following: + 1. Creates a `scheduledMetricCollectorsExecutor`. + 2. `scheduledMetricCollectorsExecutor.addScheduledMetricCollector(new XYZMetricsCollector())` is called for all collectors. + 3. `scheduledMetricCollectorsExecutor.start()` is called and then `EventLogQueueProcessor.scheduleExecutor()`. + +- Methods in `PerformanceAnalyzerPlugin` interface with the OpenSearch plugin architecture + - `onIndexModule`, `onDiscovery`, etc. are all called by OpenSearch when their corresponding events occur and the plugin can act on them. + - For example: + - `getActionFilters` provides OpenSearch with a list of classes that implement `ActionFilter`. + - `action/PerformanceAnalyzerActionFilter` is the only class currently returned to OpenSearch as an `ActionFilter`. + - when a BulkRequest or SearchRequest is recieved by OpenSearch, `action/PerformanceAnalyzerActionFilter` logs a start event and creates a listener (`action/PerformanceAnalyzerActionListener`) which waits to record the corresponding end event. + - `PerformanceAnalyzerPlugin.getRestHandlers` returns all the classes that can handle REST requests to OpenSearch. +- The classes in `http_action/config` define all the public API routes for Performance Analyzer. + - `http_action/config/RestConfig` defines `PA_BASE_URI = "/_plugins/_performanceanalyzer"` + - `PerformanceAnalyzerResourceProvider` defines the `metrics`, `rca`, `batch` and `actions` routes. + - `PerformanceAnalyzerResourceProvider.SUPPORTED_REDIRECTIONS = ("rca", "metrics", "batch", "actions")` +- `listener/PerformanceAnalyzerSearchListener` hooks into OpenSearch core to emit search operation related metrics. + + +- `writer/EventLogQueueProcessor`: + - contains `purgeQueueAndPersist` which drains `PerformanceAnalyzerMetrics.metricQueue` into a file that contains all events for a certain time bucket. It also removes old events. Uses `event_process.EventLogFileHandler` in Performance Analyzer Commons for file writing logic. + - `scheduleExecutor` periodically runs `purgeQueueAndPersist`. + +- Classes in `collectors` extend `PerformanceAnalyzerMetricsCollector` and implement `MetricsProcessor`. + - `PerformanceAnalyzerMetricsCollector` implements `Runnable` and contains common variables like `value` where a collector stores serialized metrics. + - A collector is triggered through `PerformanceAnalyzerMetricsCollector.collectMetrics`. + 1. The collector will store serialized data in the `value` variable and then call `MetricsProcessor.saveMetricValues`. + 2. `saveMetricValues` calls `PerformanceAnalyzerMetrics.emitMetric` that creates an `Event` from the serialized data and adds it to a static queue (`PerformanceAnalyzerMetrics.metricQueue`) shared across all collectors. \ No newline at end of file diff --git a/docs/READER.md b/docs/READER.md index 88a60725..eccd0dee 100644 --- a/docs/READER.md +++ b/docs/READER.md @@ -5,20 +5,18 @@ Performance Analyzer is an agent and REST API that allows you to query numerous * Performance analyzer plugin aka writer * Performance analyzer application aka reader -![alt text][performance analyzer] +![alt text](images/arch.png "Performance Analyzer Architecture") -[performance analyzer]: https://github.com/opensearch-project/performance-analyzer/blob/main/docs/images/pa.png "Performance Analyzer Architecture diagram" # Performance Analyzer plugin The performance analyzer plugin captures important events in OpenSearch and writes them to a shared memory. These events are then analyzed separately by the reader. Having separate processes, gives us better isolation between OpenSearch and metrics collection/aggregation. - # Performance Analyzer Application The performance analyzer application is written in Java, and this allows us to share code with the writer easily. Java libraries like jdbc and jooq made it very easy to generate sql programmatically. ![alt text][reader] -[reader]: https://github.com/opensearch-project/performance-analyzer/blob/main/docs/images/reader.png "Reader Architecture diagram" +[reader]: images/reader.png "Reader Architecture diagram" * MetricsProcessor - Periodically (every 5 seconds) processes all the events and statistics generated by the writer and generates metrics out of them. These metrics are then stored in metricsDB. * MetricsDB - Sqlite database that contains all the metrics that can be queried by the client. We create a new db every 5 seconds. * Webservice - A http server that serves client API requests. It services client requests by querying metricsDB for the relevant metrics requested by the client. diff --git a/docs/images/arch.png b/docs/images/arch.png new file mode 100644 index 00000000..240f9e4e Binary files /dev/null and b/docs/images/arch.png differ