Tags: microsoft/DiskANN
Tags
DiskANN v0.49.0 (#825) ## What's Changed * [quantization] 8bit distance kernels and ZipUnzip by @arkrishn94 in #798 * bumping up bf-tree version by @backurs in #819 * Import diskann-garnet and vectorset by @metajack in #800 * Saving and loading in-memory btrees to disk by @backurs in #820 ## New Contributors * @metajack made their first contribution in #800 **Full Changelog**: v0.48.0...v0.49.0 Co-authored-by: Copilot <[email protected]>
DiskANN v0.48.0 (#813) ## What's Changed * Expand BetaFilter for External SearchStrategy Implementations by @hailangx in #793 * Enable Miri tests in CI with non-blocking execution by @Copilot in #697 * Increase unit test coverage for diskann-disk crate by @Copilot in #761 * Add code review instructions for license headers by @harsha-simhadri in #801 * Consolidate `load_bin` somewhat by @hildebrandmw in #792 * Add RFC process by @suri-kumkaran in #804 * Fix variance issues with `Mat` and `MatMut`. by @hildebrandmw in #806 * [Bugfix] NeighborPriorityQueue panics, when it is at capacity and NaN distance is inserted by @arrayka in #796 * Enable `cache_only` mode for the BF-Tree cache by @hildebrandmw in #810 * Enable customization of Tokio runtimes in `diskann-benchmark-core` by @hildebrandmw in #809 ## New Contributors * @hailangx made their first contribution in #793 **Full Changelog**: v0.47.0...v0.48.0 Co-authored-by: Mark Hildebrand <[email protected]>
Bump version to 0.47.0 (#795) # DiskANN v0.47.0 ## Summary * This version contains a major breaking change to the search interface of `DiskANNIndex`. Please read the upgrade instructions below. * An Aarch64 Neon has been added to `diskann-wide`. * Various bug-fixes and code-quality improvements. ## Changes to Search The search interface has been unified around a single `index.search()` entry point using the `Search` trait. The old per-search-type methods on `DiskANNIndex` (`search`, `search_recorded`, `range_search`, `multihop_search`) have been removed and replaced by typed parameter structs that carry their own search logic. ### What Changed | Removed | Replacement | |------------------------------------------------------------|--------------------------------------------------------------| | `SearchParams` struct | `diskann::graph::search::Knn` | | `RangeSearchParams` struct | `diskann::graph::search::Range` | | `SearchParamsError` | `diskann::graph::KnnSearchError` | | `RangeSearchParamsError` | `diskann::graph::RangeSearchError` | | `index.search(&strategy, &ctx, &query, ¶ms, &mut out)` | `index.search(knn, &strategy, &ctx, &query, &mut out)` | | `index.search_recorded(..., &mut recorder)` | `index.search(RecordedKnn::new(knn, &mut recorder), ...)` | | `index.range_search(&strategy, &ctx, &query, ¶ms)` | `index.search(range, &strategy, &ctx, &query, &mut ())` | | `index.multihop_search(..., &label_eval)` | `index.search(MultihopSearch::new(knn, &label_eval), ...)` | | `index.diverse_search(...)` | `index.search(Diverse::new(knn, diverse_params), ...)` | **`flat_search`** remains an inherent method on `DiskANNIndex` Its `search_params` argument changed from `&SearchParams` to `&Knn`. ### Upgrade Instructions #### 1. k-NN Search (`search`) **Before:** ```rust use diskann::graph::SearchParams; let params = SearchParams::new(10, 100, None)?; let stats = index.search(&strategy, &ctx, &query, ¶ms, &mut output).await?; ``` **After:** ```rust use diskann::graph::{Search, search::Knn}; let params = Knn::new(10, 100, None)?; // Note: params is now the FIRST argument (moved before strategy) let stats = index.search(params, &strategy, &ctx, &query, &mut output).await?; ``` Key differences: - `SearchParams` -> `Knn` (import from `diskann::graph::search::Knn`) - `SearchParamsError` -> `KnnSearchError` (import from `diskann::graph::KnnSearchError`) - Search params moved to the **first** argument of `index.search()` - `k_value`, `l_value` fields are now private; use `.k_value()`, `.l_value()` accessors (return `NonZeroUsize`) #### 2. Recorded/Debug Search (`search_recorded`) **Before:** ```rust use diskann::graph::SearchParams; let params = SearchParams::new(10, 100, None)?; let stats = index .search_recorded(&strategy, &ctx, &query, ¶ms, &mut output, &mut recorder) .await?; ``` **After:** ```rust use diskann::graph::{Search, search::{Knn, RecordedKnn}}; let params = Knn::new(10, 100, None)?; let recorded = RecordedKnn::new(params, &mut recorder); let stats = index.search(recorded, &strategy, &ctx, &query, &mut output).await?; ``` #### 3. Range Search (`range_search`) **Before:** ```rust use diskann::graph::RangeSearchParams; let params = RangeSearchParams::new(None, 100, None, 0.5, None, 1.0, 1.0)?; let (stats, ids, distances) = index .range_search(&strategy, &ctx, &query, ¶ms) .await?; ``` **After:** ```rust use diskann::graph::{ Search, search::Range, RangeSearchOutput, }; // Simple form: let params = Range::new(100, 0.5)?; // Or full options form: let params = Range::with_options(None, 100, None, 0.5, None, 1.0, 1.0)?; // Note: output buffer is `&mut ()` — results come back in the return type let result: RangeSearchOutput<_> = index .search(params, &strategy, &ctx, &query, &mut ()) .await?; // Access results: let stats = result.stats; let ids = result.ids; // Vec<O> let distances = result.distances; // Vec<f32> ``` Key differences: - `RangeSearchParams` -> `Range` (import from `diskann::graph::search::Range`) - `RangeSearchParamsError` -> `RangeSearchError` (import from `diskann::graph::RangeSearchError`) - Return type changed from `(SearchStats, Vec<O>, Vec<f32>)` to `RangeSearchOutput<O>` (a struct with `.stats`, `.ids`, `.distances` fields) - Pass `&mut ()` as the output buffer - Field `starting_l_value` -> constructor arg `starting_l` (accessor: `.starting_l()`) - Field `initial_search_slack` -> constructor arg `initial_slack` (accessor: `.initial_slack()`) - Field `range_search_slack` -> constructor arg `range_slack` (accessor: `.range_slack()`) #### 4. Multihop / Label-Filtered Search (`multihop_search`) **Before:** ```rust use diskann::graph::SearchParams; let params = SearchParams::new(10, 100, None)?; let stats = index .multihop_search(&strategy, &ctx, &query, ¶ms, &mut output, &label_eval) .await?; ``` **After:** ```rust use diskann::graph::{Search, search::{Knn, MultihopSearch}}; let knn = Knn::new(10, 100, None)?; let params = MultihopSearch::new(knn, &label_eval); let stats = index.search(params, &strategy, &ctx, &query, &mut output).await?; ``` Key differences: - `MultihopSearch` wraps a `Knn` -> label evaluator into a single params object - The label evaluator is part of the params, not a separate argument #### 5. Flat Search (unchanged method, new param type) **Before:** ```rust use diskann::graph::SearchParams; let params = SearchParams::new(10, 100, None)?; index.flat_search(&strategy, &ctx, &query, &filter, ¶ms, &mut output).await?; ``` **After:** ```rust use diskann::graph::search::Knn; let params = Knn::new(10, 100, None)?; index.flat_search(&strategy, &ctx, &query, &filter, ¶ms, &mut output).await?; ``` Only the parameter type changed (`SearchParams` -> `Knn`). ### Import Path Changes | Old | New | |------------------------------------------|--------------------------------------------------------| | `diskann::graph::SearchParams` | `diskann::graph::search::Knn` | | `diskann::graph::RangeSearchParams` | `diskann::graph::search::Range` | | `diskann::graph::SearchParamsError` | `diskann::graph::KnnSearchError` | | `diskann::graph::RangeSearchParamsError` | `diskann::graph::RangeSearchError` | | — | `diskann::graph::search::MultihopSearch` (new) | | — | `diskann::graph::search::RecordedKnn` (new) | | — | `diskann::graph::search::Diverse` (new, feature-gated) | | — | `diskann::graph::Search` (trait, re-exported) | | — | `diskann::graph::RangeSearchOutput` (re-exported) | ## Change List * copy bftrees from the snapshot location to the save location by @backurs in #783 * (RFC) Refactor search interface with unified SearchDispatch trait by @narendatha in #773 * Make queue.closest_notvisited() safe and update call sites by @arrayka in #787 * git ignore: Ignore local settings for claude code AI agent by @arrayka in #789 * Enabling flag support in codecov by @arrayka in #790 * Increase unit test coverage for diskann-tools crate by @Copilot in #763 * Neon MVP by @hildebrandmw in #777 * Adding GraphParams to be able to save graph parameters of index to SavedParams by @backurs in #786 ## New Contributors * @narendatha made their first contribution in #773 **Full Changelog**: 0.46.0...v0.47.0
Bump version to 0.46.0 (#776) ## What's Changed ### API Breaking Changes * Remove the `experimental_avx512` feature. by @hildebrandmw in #732 * Use VirtualStorageProvider::new_overlay(test_data_root()) in tests by @Copilot in #726 * save and load max_record_size and leaf_page_size for bftrees by @backurs in #724 * [multi-vector] Verify `Standard` won't overflow in its constructor. by @hildebrandmw in #757 * VirtualStorageProvider: Make new() private, add new_physical by @Copilot in #764 * [minmax] Refactor full query by @arkrishn94 in #770 * Bump diskann-quantization to edition 2024. by @hildebrandmw in #772 ### Additions * [multi-vector] Enable cloning of `Mat` and friends. by @hildebrandmw in #759 * adding bftreepaths in mod.rs by @backurs in #775 * [quantization] Add `as_raw_ptr`. by @hildebrandmw in #774 ### Bug Fixes * Fix `diskann` compilation without default-features and add CI tests. by @hildebrandmw in #722 ### Docs and Comments * Updating the benchmark README to use diskann-benchmark by @bryantower in #709 * Fix doc comment: Windows line endings are \r\n not \n\r by @Copilot in #717 * Fix spelling errors in streaming API documentation by @Copilot in #715 * Add performance diagnostic to `diskann-benchmark` by @hildebrandmw in #744 * Add agents.md onboarding guide for coding agents by @Copilot in #765 * [doc] Fix lots of little typos in `diskann-wide` by @hildebrandmw in #771 ### Performance * [diskann-wide] Optimize `load_simd_first` for 8-bit and 16-bit element types. by @hildebrandmw in #747 ### Dependencies * Bump bytes from 1.11.0 to 1.11.1 by @dependabot[bot] in #723 * [diskann] Add note on the selection of `PruneKind` in `graph::config::Builder`. by @hildebrandmw in #734 * [diskann-providers] Remove the LRU dependency and make `vfs` and `serde_json` optional. by @hildebrandmw in #733 ### Infrastructure * Add initial QEMU tests for `diskann-wide`. by @hildebrandmw in #719 * [CI] Skip coverage for Dependabot. by @hildebrandmw in #725 * Add miri test coverage to CI workflow by @Copilot in #729 * [CI] Add minimal ARM checks by @hildebrandmw in #745 * Enable CodeQL security analysis by @Copilot in #754 ## New Contributors * @backurs made their first contribution in #724 * @arkrishn94 made their first contribution in #770 **Full Changelog**: 0.45.0...0.46.0
Version bump 0.7.0rc2->0.7.0 (#510) * Version bump 0.7.0rc2->0.7.0 Preparing diskannpy for 0.7.0 release (filter support, static memory indices only) * Update pyproject.toml the GPG key from (presumably) 2019 is no longer valid * Update pyproject.toml * Update python-release.yml By default, GITHUB_TOKEN no longer has write permissions - you have to explicitly ask for it in the specific job that needs it. We use write permissions to update the Github release action that updates the published build artifacts with the results of the release flow.
Fixing index_prefix_path bug in python for StaticMemoryIndex (#491) * Fixing the same bug I had in static disk index inside of static memory index as well. * Unit tests and a better understanding of why the unit tests were successful despite this bug
Adding Filtered Index support to Python bindings (#482) * Halfway approach to the new indexfactory, but it doesn't have the same featureset as the old way. Committing this for posterity but reverting my changes ultimately * Revert "Halfway approach to the new indexfactory, but it doesn't have the same featureset as the old way. Committing this for posterity but reverting my changes ultimately" This reverts commit 03dccb5. * Adding filtered search. API is going to change still. * Further enhancements to the new filter capability in the static memory index. * Ran automatic formatting * Fixing my logic and ensuring the unit tests pass. * Setting this up as a rc build first * list[list[Hashable]] -> list[list[str]] * Adding halfway to a solution where we query for more items than exist in the filter set. We need to replicate this behavior across all indices though - dynamic, static disk and memory w/o filters, etc * Removing the import of Hashable too
PreviousNext