Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: variable block size & elias-fano encoding #18

Open
wants to merge 7 commits into
base: main
Choose a base branch
from

Conversation

silver-ymz
Copy link
Member

@silver-ymz silver-ymz commented Dec 19, 2024

bench

  • machine: 8 vCPU, 32 GB, AMD Ryzen 7 7700, hyper-v Virtualization

dataset: trec-covid

before the PR:
Index: 4.5458s (37689.88/s)
Query: top1000 - 15.9687s (31.31/s), top10 - 18.0994s (138.13/s)
Index disk size: 384 MB

fixed block partition + delta bitpack:
Index: 4.7356s (36179.51/s)
Query: top1000 - 12.9719s (38.54/s), top10 - 13.9670s (178.99/s)
Index disk size: 508 MB

fixed block partition + elias fano:
Index: 4.8131s (35597.16/s)
Query: top1000 - 13.1613s (37.99/s), top10 - 13.3764s (186.90/s)
Index disk size: 508 MB

variable block partition + delta bitpack:
Index: 9.2832s (18456.21/s)
Query: top1000 - 13.1917s (37.90/s), top10 - 13.4416s (185.99/s)
Index disk size: 508 MB

variable block partition + elias fano:
Index: 9.3289s (18365.81/s)
Query: top1000 - 12.9883s (38.50/s), top10 - 12.4730s (200.43/s)
Index disk size: 513 MB

dataset: arguana

before the PR:
Index: 2.1008s (4128.82/s)
Query: top1000 - 47.9059s (29.35/s), top10 - 35.7517s (39.33/s)
Index disk size: 260 MB

fixed block partition + delta bitpack:
Index: 2.5946s (3343.05/s)
Query: top1000 - 36.7708s (38.24/s), top10 - 19.0610s (73.76/s)
Index disk size: 347 MB

fixed block partition + elias fano:
Index: 2.6600s (3260.88/s)
Query: top1000 - 36.0452s (39.01/s), top10 - 18.4784s (76.09/s)
Index disk size: 347 MB

variable block partition + delta bitpack:
Index: 2.7843s (3115.37/s)
Query: top1000 - 33.9380s (41.43/s), top10 - 18.7480s (74.99/s)
Index disk size: 347 MB

variable block partition + elias fano:
Index: 2.7671s (3134.68/s)
Query: top1000 - 33.4785s (42.00/s), top10 - 17.4946s (80.37/s)
Index disk size: 347 MB

@silver-ymz silver-ymz changed the title [WIP] feat: variable block size [WIP] feat: variable block size & elias-fano encoding Jan 7, 2025
Signed-off-by: Mingzhuo Yin <[email protected]>
@silver-ymz silver-ymz marked this pull request as ready for review January 7, 2025 14:57
@silver-ymz silver-ymz changed the title [WIP] feat: variable block size & elias-fano encoding feat: variable block size & elias-fano encoding Jan 7, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant