Tracks T1 T2 NeurIPS21 numbers (FALCONN-LIB#95)

* added csv files for T1, T2 results * added link from README to t1.csv and t2.csv * added T1, T2 plots for neurips results * remove results/ folder
duckth · Feb 1, 2022 · c35a05d · c35a05d
1 parent c182824
commit c35a05d
Show file tree

Hide file tree

Showing 21 changed files with 479 additions and 60 deletions.
diff --git a/results/bigann-1B.png b/results/bigann-1B.png
diff --git a/results/deep-1B.png b/results/deep-1B.png
diff --git a/results/msspacev-1B.png b/results/msspacev-1B.png
diff --git a/results/msturing-1B.png b/results/msturing-1B.png
diff --git a/results/text2image-1B.png b/results/text2image-1B.png
diff --git a/t1_t2/README.md b/t1_t2/README.md
@@ -266,6 +266,8 @@ Algorithms will be ranked on how much their recall surpasses the baselines at th
 #### Results for T1
 The table lists best Recall/AP obtained at at least 10,000 QPS by algorithms in pull requests submitted before Nov 2021. All non-empty cells are derived from author-published indices that succesfully ran on the standardized hardware with author-provided query configurations. Recall for bigann-1B does not count ties. This result is on the public query set.
 
+A complete set of runs can be found [here](results/T1/neurips21/t1.csv) and QPS vs recall plots are [here](results/T1/neurips21/)
+
 
 | PR  | Name      |bigann-1B  | deep-1B |	msspacev-1B |	msturing-1B |	ssnpp-1B  |	text2image-1B  |
 |-----|-----------|-----------|---------|-------------|-------------|-----------|----------------|
@@ -283,6 +285,7 @@ The table lists best Recall/AP obtained at at least 10,000 QPS by algorithms in
 #### Results for T2
 The table lists best Recall/AP obtained at at least 1,500 QPS by algorithms in pull requests submitted before Nov 2021. All non-empty cells are derived from author-published indices that succesfully ran on the standardized hardware with author-provided query configurations. Recall for bigann-1B does not count ties. This result is on the public query set.
 
+A complete set of runs can be found [here](results/T2/neurips21/t2.csv) and QPS vs recall plots are [here](results/T2/neurips21/)
 
 | PR | Name      |bigann-1B  | deep-1B |	msspacev-1B |	msturing-1B |	ssnpp-1B  |	text2image-1B |
 |----|-----------|-----------|---------|-------------|-------------|-----------|----------------|

diff --git a/t1_t2/results/T1/neurips21/bigann-1B.png b/t1_t2/results/T1/neurips21/bigann-1B.png
diff --git a/t1_t2/results/T1/neurips21/deep-1B.png b/t1_t2/results/T1/neurips21/deep-1B.png
diff --git a/t1_t2/results/T1/neurips21/msspacev-1B.png b/t1_t2/results/T1/neurips21/msspacev-1B.png
diff --git a/t1_t2/results/T1/neurips21/msturing-1B.png b/t1_t2/results/T1/neurips21/msturing-1B.png
diff --git a/results/ssnpp-1B.png → t1_t2/results/T1/neurips21/ssnpp-1B.png b/results/ssnpp-1B.png → t1_t2/results/T1/neurips21/ssnpp-1B.png
diff --git a/t1_t2/results/T1/neurips21/t1.csv b/t1_t2/results/T1/neurips21/t1.csv
diff --git a/t1_t2/results/T1/neurips21/text2image-1B.png b/t1_t2/results/T1/neurips21/text2image-1B.png
diff --git a/t1_t2/results/T2/neurips21/bigann-1B.png b/t1_t2/results/T2/neurips21/bigann-1B.png
diff --git a/t1_t2/results/T2/neurips21/deep-1B.png b/t1_t2/results/T2/neurips21/deep-1B.png
diff --git a/t1_t2/results/T2/neurips21/msspacev-1B.png b/t1_t2/results/T2/neurips21/msspacev-1B.png
diff --git a/t1_t2/results/T2/neurips21/msturing-1B.png b/t1_t2/results/T2/neurips21/msturing-1B.png
diff --git a/t1_t2/results/T2/neurips21/ssnpp-1B.png b/t1_t2/results/T2/neurips21/ssnpp-1B.png
diff --git a/t1_t2/results/T2/neurips21/t2.csv b/t1_t2/results/T2/neurips21/t2.csv
@@ -0,0 +1,65 @@
+algorithm,parameters,dataset,count,qps,distcomps,build,indexsize,queriessize,mean_ssd_ios,mean_latency,recall/ap
+kota-t2,DiskANN,bigann-1B,10,1729.9645217994996,2268.1108,1000000.0,60379620.0,34902.23021290255,55.7439,4592.0382,0.9371
+kota-t2,DiskANN,bigann-1B,10,1313.924291806509,2863.43984,1000000.0,60379620.0,45953.652258749484,74.3488,6001.7167,0.9640000000000001
+kota-t2,DiskANN,bigann-1B,10,1157.3305547640223,3206.66504,1000000.0,60379620.0,52171.455900351044,85.1725,6820.1601,0.9731
+kota-t2,DiskANN,bigann-1B,10,1420.3258344842309,2677.71058,1000000.0,60379620.0,42511.10451844025,68.5255,5564.5106,0.9581500000000001
+kota-t2,DiskANN,bigann-1B,10,1642.9486573844054,2375.40114,1000000.0,60379620.0,36750.76499111366,59.1014,4829.8101,0.94337
+kota-t2,DiskANN,bigann-1B,10,2217.5682863981187,1801.02192,1000000.0,60379620.0,27227.85150308561,41.2181,3572.5136,0.89177
+kota-t2,DiskANN,bigann-1B,10,1489.4038761039308,2580.0169,1000000.0,60379620.0,40539.45405187512,65.4768,5320.1737,0.9541000000000001
+kota-t2,DiskANN,bigann-1B,10,1564.4039965126092,2479.1623200000004,1000000.0,60379620.0,38595.92543524504,62.3378,5068.6112,0.94878
+kota-t2,DiskANN,bigann-1B,10,1938.0611173226125,2043.5712,1000000.0,60379620.0,31154.65217289591,48.7581,4092.3886,0.9196
+kota-t2,DiskANN,bigann-1B,10,1533.9216597336165,2519.14818,1000000.0,60379620.0,39362.90984409571,63.5898,5166.4773,0.9508599999999999
+kota-t2,DiskANN,ssnpp-1B,60000,2237.9455403144407,1572.170728,1000000.0,63001892.0,28151.664490972365,31.02686,2363.41259,0.1557059316298228
+kota-t2,DiskANN,ssnpp-1B,60000,2653.424797101891,1589.73445,1000000.0,63006140.0,23745.214135639428,36.27968,2319.8051,0.14839052820176385
+kota-t2,DiskANN,ssnpp-1B,60000,3155.3199596933705,1325.716972,1000000.0,63006140.0,19968.225347936776,31.16975,1880.63766,0.13241077434436713
+kota-t2,DiskANN,ssnpp-1B,60000,1968.8022474863317,1849.401862,1000000.0,63001892.0,32000.111783922264,37.09872,2799.77005,0.1821203017821256
+bbann,BBANN,text2image-1B,10,1902.735310390915,0.0,1000000.0,29064836.0,15275.29123009162,0.0,0.0,0.45160600000000006
+bbann,BBANN,text2image-1B,10,1592.557788590405,0.0,1000000.0,29064836.0,18250.412140915585,0.0,0.0,0.49080700000000005
+bbann,BBANN,text2image-1B,10,2197.8065625321065,0.0,1000000.0,29064836.0,13224.474116827745,0.0,0.0,0.42842900000000006
+bbann,BBANN,text2image-1B,10,1540.622672933968,0.0,1000000.0,29064836.0,18865.64212679592,0.0,0.0,0.495423
+bbann,BBANN,text2image-1B,10,1571.8388035067858,0.0,1000000.0,29064836.0,18490.97753227373,0.0,0.0,0.486085
+bbann,BBANN,text2image-1B,10,1581.271710631964,0.0,1000000.0,29064836.0,18380.671585140848,0.0,0.0,0.49324500000000004
+bbann,BBANN,text2image-1B,10,1608.3394385279835,0.0,1000000.0,29064836.0,18071.332023419945,0.0,0.0,0.484772
+bbann,BBANN,text2image-1B,10,1555.2814633486682,0.0,1000000.0,29064836.0,18687.830264125092,0.0,0.0,0.487202
+bbann,BBANN,text2image-1B,10,2314.605123985524,0.0,1000000.0,29064836.0,12557.146659190483,0.0,0.0,0.425323
+bbann,BBANN,text2image-1B,10,1685.0312317828493,0.0,1000000.0,29064836.0,17248.841120439007,0.0,0.0,0.471499
+kota-t2,DiskANN,msturing-1B,10,1869.9105083966315,2199.94934,1000000.0,60658012.0,32438.99198791693,55.69439,4245.34491,0.926499
+kota-t2,DiskANN,msturing-1B,10,1708.9818447326425,2377.74471,1000000.0,60658012.0,35493.65500105093,61.39347,4646.6233,0.934886
+kota-t2,DiskANN,msturing-1B,10,1829.7687123162966,2242.62236,1000000.0,60658012.0,33150.64444577439,57.06105,4338.58571,0.928661
+kota-t2,DiskANN,msturing-1B,10,1606.1513481600634,2508.44176,1000000.0,60658012.0,37766.062376056376,65.60437,4942.56754,0.939817
+kota-t2,DiskANN,msturing-1B,10,2151.6371656304373,1945.2767220000003,1000000.0,60658012.0,28191.561741417954,47.59611,3688.2624,0.909615
+kota-t2,DiskANN,msturing-1B,10,1840.9913542224767,2228.411372,1000000.0,60658012.0,32948.558862525606,56.60407,4312.21896,0.927989
+kota-t2,DiskANN,msturing-1B,10,1898.1747241126752,2171.368234,1000000.0,60658012.0,31955.968662661086,54.78257,4182.6914,0.9249430000000001
+kota-t2,DiskANN,msturing-1B,10,2384.73537024552,1774.407974,1000000.0,60658012.0,25435.950989293615,42.20301,3325.88183,0.892944
+kota-t2,DiskANN,msturing-1B,10,1974.2327081034416,2098.642458,1000000.0,60658012.0,30724.85414258559,52.46636,4021.74299,0.92079
+kota-t2,DiskANN,msturing-1B,10,1441.516326219669,2758.310464,1000000.0,60658012.0,42079.30974952862,73.68436,5507.78782,0.947376
+kota-t2,DiskANN,msspacev-1B,10,1375.2978140778016,2810.365493246009,1000000.0,57428288.0,41756.983405451145,82.98396779915404,5785.338279437849,0.9073441124300723
+kota-t2,DiskANN,msspacev-1B,10,1538.8314288346342,2547.252121708282,1000000.0,57428288.0,37319.41454009083,73.47434847864648,5165.493246009005,0.9040012279983627
+kota-t2,DiskANN,msspacev-1B,10,1637.764743606222,2414.896950470733,1000000.0,57428288.0,35065.04107149582,68.6947400736799,4853.005628325829,0.9019068085686996
+kota-t2,DiskANN,msspacev-1B,10,1894.0193603989806,2130.699065356802,1000000.0,57428288.0,30320.855848011273,58.47274525856188,4193.0943170964665,0.8951801064265247
+kota-t2,DiskANN,msspacev-1B,10,1244.4963678400923,3068.2946650293356,1000000.0,57428288.0,46145.80603370557,92.29093327875563,6388.655137126484,0.9099501978441806
+kota-t2,DiskANN,msspacev-1B,10,1409.6927009906594,2748.5545094828763,1000000.0,57428288.0,40738.16084856108,80.73986901350798,5638.904489016237,0.906951835175331
+kota-t2,DiskANN,msspacev-1B,10,1492.545081361286,2617.2697025515076,1000000.0,57428288.0,38476.75270727644,75.98819757129213,5325.529062627917,0.9050859598853869
+kota-t2,DiskANN,msspacev-1B,10,1753.873700033979,2274.3819006685767,1000000.0,57428288.0,32743.685020698696,63.63559148587802,4529.054577705007,0.8993280120070951
+kota-t2,DiskANN,msspacev-1B,10,1451.1867197104223,2681.414306180925,1000000.0,57428288.0,39573.327966686156,78.30474826033566,5475.296015827535,0.9058568699686178
+kota-t2,DiskANN,msspacev-1B,10,1305.3203479806382,2940.7762586983217,1000000.0,57428288.0,43995.55104526099,87.67921953881839,6087.747373447946,0.9086164551780598
+bbann,BBANN,msspacev-1B,10,1610.4208408102975,0.0,1000000.0,11878352.0,7375.930377318827,0.0,0.0,0.7577705007504434
+bbann,BBANN,msspacev-1B,10,1820.4236647258076,0.0,1000000.0,11878352.0,6525.048113890081,0.0,0.0,0.747574703233729
+bbann,BBANN,msspacev-1B,10,1755.869157475597,0.0,1000000.0,11878352.0,6764.941424836825,0.0,0.0,0.7542058943921408
+bbann,BBANN,msspacev-1B,10,1936.7360372815363,0.0,1000000.0,11878352.0,6133.1806561893845,0.0,0.0,0.7458589166325555
+bbann,BBANN,msspacev-1B,10,3912.993335161897,0.0,1000000.0,11878352.0,3035.617743905138,0.0,0.0,0.6773809523809524
+bbann,BBANN,msspacev-1B,10,1821.776089651613,0.0,1000000.0,11878352.0,6520.20413895736,0.0,0.0,0.7500818665575113
+bbann,BBANN,msspacev-1B,10,1518.3505425395547,0.0,1000000.0,11878352.0,7823.1947545739795,0.0,0.0,0.7602026197298404
+bbann,BBANN,msspacev-1B,10,1702.433482388304,0.0,1000000.0,11878352.0,6977.2781861269195,0.0,0.0,0.7522240414790559
+bbann,BBANN,msspacev-1B,10,1629.0401932767472,0.0,1000000.0,11878352.0,7291.626105373854,0.0,0.0,0.7563787692727522
+bbann,BBANN,msspacev-1B,10,2070.9883234427366,0.0,1000000.0,11878352.0,5735.595833903039,0.0,0.0,0.7408104789193615
+bbann,BBANN,ssnpp-1B,96237,1462.0503482699048,0.0,1000000.0,41172708.0,28160.937172048216,0.0,0.0,0.8863012589479978
+bbann,BBANN,ssnpp-1B,96237,1671.0205269554601,0.0,1000000.0,41172708.0,24639.259264525735,0.0,0.0,0.8772537380208939
+bbann,BBANN,ssnpp-1B,96237,1516.7019615432832,0.0,1000000.0,41172708.0,27146.208710711835,0.0,0.0,0.8857355212675846
+bbann,BBANN,ssnpp-1B,96237,1587.3311664137136,0.0,1000000.0,41172708.0,25938.322683490336,0.0,0.0,0.8816703846185786
+bbann,BBANN,ssnpp-1B,96237,1707.1922937796771,0.0,1000000.0,41172708.0,24117.2058648676,0.0,0.0,0.872364908070222
+bbann,BBANN,ssnpp-1B,96237,1545.1303206752962,0.0,1000000.0,41172708.0,26646.75428931169,0.0,0.0,0.8824276426697005
+bbann,BBANN,ssnpp-1B,96237,1455.0100602629188,0.0,1000000.0,41172708.0,28297.19815996333,0.0,0.0,0.8861660528326329
+bbann,BBANN,ssnpp-1B,96237,1767.7956237172189,0.0,1000000.0,41172708.0,23290.42308263237,0.0,0.0,0.8772940922453454
+bbann,BBANN,ssnpp-1B,96237,1380.1362515617418,0.0,1000000.0,41172708.0,29832.350214270202,0.0,0.0,0.8889190932833961
+bbann,BBANN,ssnpp-1B,96237,1905.0949488596978,0.0,1000000.0,41172708.0,21611.892900479364,0.0,0.0,0.8731373476141441
diff --git a/t1_t2/results/T2/neurips21/text2image-1B.png b/t1_t2/results/T2/neurips21/text2image-1B.png
diff --git a/t1_t2/results/T2/res.csv b/t1_t2/results/T2/res.csv