milvus - VectorDBBench benchmaker 性能测试工具使用经验

IVF_FLAT (Inverted File with Flat Indexing)
优点:
在数据量适中且维度不是非常高的情况下，IVF_FLAT能提供精确的最近邻搜索结果。
相对简单，易于理解和实现。
缺点:
当数据集非常大时，IVF_FLAT需要大量的内存来存储整个数据集，这可能导致性能瓶颈。
检索速度随着数据集的增长而显著降低，因为需要与所有候选数据点计算相似度。
HNSW (Hierarchical Navigable Small World Graphs)
优点:
HNSW特别适合高维数据，能够在大规模数据集上提供高效的近似最近邻搜索。
即使数据集很大，HNSW也能保持较快的查询速度。
缺点:
HNSW的构建过程可能较为复杂和耗时，尤其是在数据流式更新的情况下。
结果为近似最近邻，虽然通常足够准确，但在某些场景下可能不如IVF_FLAT精确。
对于小数据场景，特别是如果数据量不大且可以接受较高的内存消耗，IVF_FLAT可能是一个更好的选择，因为它能提供精确的结果。然而，如果数据集的维度非常高或者你希望优化查询速度，即使数据量较小，HNSW仍然可能是更优的选择。
在实际应用中，选择哪一种算法还应考虑具体业务需求，比如是否可以容忍近似结果、系统资源限制、以及是否需要实时更新索引等因素。如果数据量和维度都不大，而且对查询速度的要求不是特别高，那么IVF_FLAT可能就足够了；但如果需要在高维数据上进行快速查询，HNSW则更加合适。

HNSW  分层可导航小世界图索引
1.M
M是一个关键参数，它定义了图中每个节点的最大出边数，即每个节点可以连接到的最多其他节点的数量 
在HNSW的多级图结构中，M控制着图的连通性和密度。
2.efConstruction
efConstruction定义了在构建图时，算法为每个节点寻找最近邻时考虑的候选节点数量。换句话说，它是构建索引过程中用于决定每个节点连接哪些其他节点的探索因子。
它控制了算法在添加新节点时考虑的候选邻居数量。当添加一个新节点时，算法会从当前图中选取efConstruction个节点作为候选，并从中挑选出最佳的邻居节点来建立连接
3.ef-search 
ef_search（有时简写为 ef）是HNSW（Hierarchical Navigable Small World graphs）算法中用于查询阶段的一个重要参数。在查询时，ef_search定义了算法在试图找到最近邻时考虑的候选节点数量。
它控制着搜索的广度，即算法在搜索图时愿意检查多少条边。
ef_search 控制搜索过程的效率和精度，它影响算法在搜索时探索的路径数量。
4.k
k通常是指查询时返回的最近邻（nearest neighbors）的数量

IVF_FALT  带平面索引的倒排文件
lists
lists参数实际上指的是nlist，它是IVF_FLAT索引中聚类（或列表）的数量。nlist决定了数据集被分割成多少个子集。
probes
probes参数是指在搜索时实际检查的聚类数量

官网地址如下：
https://zilliz.com/vector-database-benchmark-tool?database=ZillizCloud%2CMilvus%2CElasticCloud%2CPgVector%2CPinecone%2CQdrantCloud%2CWeaviateCloud&dataset=medium&filter=none%2Clow%2Chigh&tab=1

https://gitee.com/woohello/VectorDBBench

1.前置条件：
1.python >= 3.11
Ssl 问题解决：
https://cloud.tencent.com/developer/article/2424089
https://blog.csdn.net/qq_43370607/article/details/137382955

pip install vectordb-bench

1.ivf_flat 性能验证

nohup ./ivf_flat.sh  > /root/octopus/benchmarker/dimension/ivf_flat.log  2>&1 &

/usr/local/python3.12/bin/vectordbbench  milvusivfflat --uri 'http://xxxxx:32437'  --drop-old --lists 1024 --probes 5

{"run_id": "09c01adfa7a5470ea6176855cccd5628","task_label": "09c01adfa7a5470ea6176855cccd5628","results": [{"metrics": {"max_load_count": 0,"load_duration": 158.9442,"qps": 3031.743,"serial_latency_p99": 0.008,"recall": 0.5065,"ndcg": 0.5493,"conc_num_list": [1, 5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100],"conc_qps_list": [175.7321, 838.1756, 1328.4196, 1676.7759, 1885.2717, 2329.0303, 2238.8839, 2307.6002, 2418.9697, 2502.9213, 2628.4562, 2611.9513, 2631.2078, 2709.1865, 2751.1792, 2749.1104, 2792.9155, 2752.8945, 2836.4445, 2904.4609, 3031.743],"conc_latency_p99_list": [0.004655693072453141, 0.004652024507522583, 0.004790452426299453, 0.005058524944633246, 0.0053853894943371415, 0.00560504300147295, 0.005771844685822725, 0.006148010149598122, 0.006257654917612672, 0.00643803995847702, 0.006523223061859608, 0.00653654379490763, 0.0069596841046586635, 0.00731352926902473, 0.007371303759515286, 0.007228717562556266, 0.007472157787531615, 0.0074205820478498935, 0.006961745805479586, 0.008283554406836629, 0.0073261844931170345]},"task_config": {"db": "Milvus","db_config": {"db_label": "2024-07-30T14:43:55.771839","uri": "**********"},"db_case_config": {"index": "IVF_FLAT","metric_type": "COSINE","nlist": 1024,"nprobe": 5},"case_config": {"case_id": 50,"custom_case": null,"k": 100,"concurrency_search_config": {"num_concurrency": [1, 5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100],"concurrency_duration": 30}},"stages": ["drop_old", "load", "search_serial", "search_concurrent"]},"label": ":)"}],"file_fmt": "result_{}_{}_{}.json"
}

2.hnsw 性能验证
条件：–m 8 --ef-construction 64 --ef-search 10 --k 10

nohup ./hnsw_10.sh > /root/octopus/benchmarker/dimension/hnsw_10.log  2>&1 & 

/usr/local/python3.12/bin/vectordbbench   milvushnsw --uri 'http://172.20.1.101:32437' --drop-old   --m 8 --ef-construction 64  --ef-search 10  --k 10

{"run_id": "1c945b28090941c7ae58ec55a8ae0ecb","task_label": "1c945b28090941c7ae58ec55a8ae0ecb","results": [{"metrics": {"max_load_count": 0,"load_duration": 138.8369,"qps": 3857.9497,"serial_latency_p99": 0.006,"recall": 0.6946,"ndcg": 0.7127,"conc_num_list": [1, 5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100],"conc_qps_list": [205.9066, 896.0748, 1535.6513, 1961.4574, 2282.5936, 2486.9112, 2473.0187, 2639.9002, 2792.597, 2825.9357, 2985.0966, 2992.8825, 3202.8998, 3336.8242, 3480.6602, 3195.9544, 3206.5109, 3249.2361, 3527.203, 3528.9396, 3857.9497],"conc_latency_p99_list": [0.004022291118651628, 0.003988313633389771, 0.004059490236081183, 0.0042587881341576575, 0.004422521588951349, 0.0045851776339113715, 0.0050087556691840295, 0.005091396714560687, 0.005392795985937118, 0.005527270000986755, 0.00542353574000299, 0.005898943385109305, 0.005861698154173791, 0.006148975464701652, 0.006199899067729712, 0.006067702007293701, 0.006825755665823817, 0.006885729782283306, 0.006695127539336682, 0.0061551205324009055, 0.006313137001730501]},"task_config": {"db": "Milvus","db_config": {"db_label": "2024-07-30T15:32:41.701653","uri": "**********"},"db_case_config": {"index": "HNSW","metric_type": "COSINE","M": 8,"efConstruction": 64,"ef": 10},"case_config": {"case_id": 50,"custom_case": null,"k": 10,"concurrency_search_config": {"num_concurrency": [1, 5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100],"concurrency_duration": 30}},"stages": ["drop_old", "load", "search_serial", "search_concurrent"]},"label": ":)"}],"file_fmt": "result_{}_{}_{}.json"
}

2.hnsw 性能验证
条件： --m 8 --ef-construction 64 --ef-search 64 --k 64

nohup ./hnsw_10.sh > /root/octopus/benchmarker/dimension/hnsw_64.log 2>&1 & 

{"run_id": "0112f04d884a48e7b04a79601b501f43","task_label": "0112f04d884a48e7b04a79601b501f43","results": [{"metrics": {"max_load_count": 0,"load_duration": 99.8713,"qps": 3590.8011,"serial_latency_p99": 0.0068,"recall": 0.6842,"ndcg": 0.6988,"conc_num_list": [1, 5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100],"conc_qps_list": [204.2923, 903.9947, 1569.658, 1960.1058, 2254.7686, 2555.6955, 2661.9067, 2884.7484, 2974.3032, 2975.3197, 2988.5105, 3178.9354, 3299.7453, 3396.6382, 3327.4601, 3397.9266, 3465.914, 3466.3555, 3471.6151, 3504.3056, 3590.8011],"conc_latency_p99_list": [0.004091060114465654, 0.003925692438334226, 0.004029340751469135, 0.004241374858655035, 0.0044741668567061426, 0.004661575911566615, 0.004779618114605546, 0.00496285007596016, 0.005089158420264721, 0.005570173567533493, 0.005635122420825064, 0.0056590593174099925, 0.005516276902705431, 0.0058326821969822045, 0.005921696238219737, 0.006209327307716011, 0.006015772367082536, 0.0063866944853216406, 0.0059822171421721575, 0.005861425867676735, 0.00654305313937366]},"task_config": {"db": "Milvus","db_config": {"db_label": "2024-07-30T16:03:25.006315","uri": "**********"},"db_case_config": {"index": "HNSW","metric_type": "COSINE","M": 8,"efConstruction": 64,"ef": 10},"case_config": {"case_id": 50,"custom_case": null,"k": 10,"concurrency_search_config": {"num_concurrency": [1, 5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100],"concurrency_duration": 30}},"stages": ["drop_old", "load", "search_serial", "search_concurrent"]},"label": ":)"}],"file_fmt": "result_{}_{}_{}.json"
}