Skip to content

Commit

Permalink
refine readme
Browse files Browse the repository at this point in the history
  • Loading branch information
lqhl committed Sep 12, 2023
1 parent bbbce4c commit 2de5ae5
Show file tree
Hide file tree
Showing 4 changed files with 24 additions and 3 deletions.
10 changes: 8 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,9 +11,15 @@ This benchmark assesses the performance of **fully-managed** vector databases wi
Here's a preview of the results:

1. **Queries Per Second (QPS):** Higher QPS is preferable as it signifies greater throughput.
- Throughput for Vector Search
![Throughput](images/qps.png)
- Throughput for Filtered Vector Search
![Throughput](images/qps-filtered-search.png)
2. The **cost-performance ratio** is calculated by dividing the monthly cost by the QPS of the services per one hundred units. A lower ratio suggests better cost effectiveness.
- Cost-performance ratio for Vector Search
![Monthly Cost ($) Per 100 QPS](images/cost-per-100-qps.png)
- Cost-performance ratio for Filtered Vector Search
![Monthly Cost ($) Per 100 QPS](images/cost-per-100-qps-filtered-search.png)

## Run the Benchmark

Expand All @@ -23,8 +29,8 @@ First, install the necessary libraries on the client used for the benchmark.
pip install -r requirements.txt
```

Afterwards, follow the [Step-by-Step Guide for Benchmark](docs/step-by-step-guide-for-benchmark.md) to execute the benchmark for each cloud service. You can refer to [Results Visualize](docs/results-visualize.md) for visualizing the test results.
Afterwards, follow the [Step-by-Step Guide for Benchmark](docs/step-by-step-guide-for-benchmark.md) to execute the benchmark for each cloud service. You can refer to [Results Visualization](docs/results-visualization.md) for visualizing the test results.

## Credit
## Special Thanks

This repository is a fork of [qdrant/vector-db-benchmark](https://github.com/qdrant/vector-db-benchmark/), specifically tailored for fully-managed vector databases.
17 changes: 16 additions & 1 deletion docs/results-visualize.md → docs/results-visualization.md
Original file line number Diff line number Diff line change
@@ -1,27 +1,42 @@
# Result Visualization

> [!NOTE]
> Before proceeding with result visualization, you need to test and obtain some **result files** according to the process in [step-by-step-guide-for-benchmark.md](./step-by-step-guide-for-benchmark.md)
## Preparation

### Gather Test Results

![MyScaleResults.jpg](../images/MyScaleResuts.jpg)

### Clone Our Benchmark Repo

```shell
git clone [email protected]:myscale/benchmark.git
```

## Handle Test Results

### Step1. Update Monthly Cost

The test results do not include the price of the cluster you tested, so you need to manually add the monthly price to the test results.
You can modify the specific price information within the [script](../scripts/handle_price.py)

```shell
cd vector-db-benchmark/scripts
python3 handle_price.py
```

### Step2. Generate `benchmark.json`

```shell
cd vector-db-benchmark/scripts
python3 process_result_files.py
```

Then you will get a json file named `benchmark.json`, please replace [old_benchmark.json](https://github.com/myscale/benchmark/blob/main/public/benchmark.json) with your new `benchmark.json`.
## View Plotly Chart

## View Plotly Chart

Please start your website according to the [readme](https://github.com/myscale/benchmark/blob/main/README.md), each result file corresponds one-to-one with a point in the scatter plot.
![ResultsVisualizePlotly.png](../images/ResultsVisualizePlotly.png)
Binary file added images/cost-per-100-qps-filtered-search.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added images/qps-filtered-search.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit 2de5ae5

Please sign in to comment.