benchmark: Benchmark laion 100M for 0.2 release candidate

[VoVAllen]

With #150 and #145
test fp16

[cutecutecat]

before: result from the last experiment at v0.1.0

now: at branch #157 as v0.2.0

test machine = i4i.xlarge -> [vcpu = 4 memory = 32G ssd = 937G]
nlist = 160000
shared buffer = 28G
jit = false
effective_io_concurrency = 200
with prewarm

Float32 laion-100m

disk usage: 840G -> 423G

build time:

• 28h -> 32h when using 8c
• 110h when using 4c

nprob / epsilon / top	QPS/Recall before	P50/P99 before	QPS/Recall now	P50/P99 now
100 / 1.5 / 10	7.24 / 0.9650	77.22 / 1042.41	4.26 / 0.9644	107.62 / 1772.86
100 / 1.0 / 10	14.84 / 0.9536	34.78 / 363.49	7.27 / 0.9531	59.77 / 841.97
100 / 0.5 / 10	25.69 / 0.9079	22.59 / 148.25	14.83 / 0.9073	31.79 / 323.03
400 / 0.8 / 10	16.18 / 0.9535	37.10 / 302.29	10.22 / 0.9530	50.86 / 513.14

Float16 laion-100m

disk usage: <210G

In progress...

Conclusions

• The PR significantly reduced disk usage
• P99 and QPS has a nearly same decrease 50%, but not related to the PR itself

[VoVAllen]

laion-100m + 2 layer IVF on main

[cutecutecat]

Comparasion of laion-100m

on v0.1.0

nprob / epsilon / top	QPS/Recall + Mem	P50 /P99 + Mem	QPS/Recall + Disk	P50 /P99 + Disk
100 / 1.5 / 10	7.09 / 0.9645	88.14 / 842.91	7.53 / 0.9645	73.56 / 1005.50
100 / 1.0 / 10	16.81 / 0.9534	34.06 / 325.99	15.29 / 0.9534	34.01 / 354.14
100 / 0.5 / 10	97.31 / 0.9077	9.84 / 14.29	26.25 / 0.9077	22.13 / 143.24
400 / 0.8 / 10	47.93 / 0.9532	20.24 / 28.37	15.87 / 0.9532	38.24 / 307.71

postgres=# SELECT vchordrq_prewarm('laion_embedding_idx'::regclass);
      vchordrq_prewarm       
-----------------------------
 height of root: 2          +
 number of tuples: 1        +
 number of pages: 1         +
 number of tuples: 160000   +
 number of pages: 2500      +
 number of tuples: 100000000+
 number of pages: 3203564   +

on #157

nprob / epsilon / top	QPS/Recall + Mem	P50 /P99 + Mem	QPS/Recall + Disk	P50 /P99 + Disk
100 / 1.5 / 10	41.96 / 0.9644	19.15 / 91.95	4.12 / 0.9644	109.98 / 1756.06
100 / 1.0 / 10	50.42 / 0.9531	17.27 / 48.18	6.97 / 0.9531	62.55 / 855.35
100 / 0.5 / 10	55.73 / 0.9073	16.28 / 28.22	12.22 / 0.9073	37.09 / 378.12
400 / 0.8 / 10	27.58 / 0.9530	33.80 / 59.21	7.49 / 0.9530	67.35 / 674.66

postgres=# SELECT vchordrq_prewarm('laion_embedding_idx'::regclass);
      vchordrq_prewarm       
-----------------------------
 height of root: 2          +
 number of tuples: 1        +
 number of pages: 1         +
 number of tuples: 160000   +
 number of pages: 2759      +
 number of tuples: 100000000+
 number of pages: 1719449   +

[VoVAllen]

Flame graph on build process