diff --git a/README.md b/README.md index a637f06..e104883 100644 --- a/README.md +++ b/README.md @@ -11,9 +11,15 @@ This benchmark assesses the performance of **fully-managed** vector databases wi Here's a preview of the results: 1. **Queries Per Second (QPS):** Higher QPS is preferable as it signifies greater throughput. + - Throughput for Vector Search ![Throughput](images/qps.png) + - Throughput for Filtered Vector Search + ![Throughput](images/qps-filtered-search.png) 2. The **cost-performance ratio** is calculated by dividing the monthly cost by the QPS of the services per one hundred units. A lower ratio suggests better cost effectiveness. + - Cost-performance ratio for Vector Search ![Monthly Cost ($) Per 100 QPS](images/cost-per-100-qps.png) + - Cost-performance ratio for Filtered Vector Search + ![Monthly Cost ($) Per 100 QPS](images/cost-per-100-qps-filtered-search.png) ## Run the Benchmark @@ -23,8 +29,8 @@ First, install the necessary libraries on the client used for the benchmark. pip install -r requirements.txt ``` -Afterwards, follow the [Step-by-Step Guide for Benchmark](docs/step-by-step-guide-for-benchmark.md) to execute the benchmark for each cloud service. You can refer to [Results Visualize](docs/results-visualize.md) for visualizing the test results. +Afterwards, follow the [Step-by-Step Guide for Benchmark](docs/step-by-step-guide-for-benchmark.md) to execute the benchmark for each cloud service. You can refer to [Results Visualization](docs/results-visualization.md) for visualizing the test results. -## Credit +## Special Thanks This repository is a fork of [qdrant/vector-db-benchmark](https://github.com/qdrant/vector-db-benchmark/), specifically tailored for fully-managed vector databases. diff --git a/docs/results-visualize.md b/docs/results-visualization.md similarity index 96% rename from docs/results-visualize.md rename to docs/results-visualization.md index 4afea12..1482697 100644 --- a/docs/results-visualize.md +++ b/docs/results-visualization.md @@ -1,27 +1,42 @@ # Result Visualization + +> [!NOTE] > Before proceeding with result visualization, you need to test and obtain some **result files** according to the process in [step-by-step-guide-for-benchmark.md](./step-by-step-guide-for-benchmark.md) + ## Preparation + ### Gather Test Results + ![MyScaleResults.jpg](../images/MyScaleResuts.jpg) + ### Clone Our Benchmark Repo + ```shell git clone git@github.com:myscale/benchmark.git ``` ## Handle Test Results + ### Step1. Update Monthly Cost + The test results do not include the price of the cluster you tested, so you need to manually add the monthly price to the test results. You can modify the specific price information within the [script](../scripts/handle_price.py) + ```shell cd vector-db-benchmark/scripts python3 handle_price.py ``` + ### Step2. Generate `benchmark.json` + ```shell cd vector-db-benchmark/scripts python3 process_result_files.py ``` + Then you will get a json file named `benchmark.json`, please replace [old_benchmark.json](https://github.com/myscale/benchmark/blob/main/public/benchmark.json) with your new `benchmark.json`. -## View Plotly Chart + +## View Plotly Chart + Please start your website according to the [readme](https://github.com/myscale/benchmark/blob/main/README.md), each result file corresponds one-to-one with a point in the scatter plot. ![ResultsVisualizePlotly.png](../images/ResultsVisualizePlotly.png) diff --git a/images/cost-per-100-qps-filtered-search.png b/images/cost-per-100-qps-filtered-search.png new file mode 100644 index 0000000..5b71463 Binary files /dev/null and b/images/cost-per-100-qps-filtered-search.png differ diff --git a/images/qps-filtered-search.png b/images/qps-filtered-search.png new file mode 100644 index 0000000..425ed88 Binary files /dev/null and b/images/qps-filtered-search.png differ