Skip to content

Commit

Permalink
doc: updates README and Index documentation
Browse files Browse the repository at this point in the history
  • Loading branch information
@arindas
arindas committed Feb 9, 2024
1 parent 1d994b2 commit b1de935
Show file tree
Hide file tree
Showing 2 changed files with 71 additions and 43 deletions.
88 changes: 47 additions & 41 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,12 @@ order they need.
Find examples demonstrating different capabilities of `laminarmq` in the
[examples branch](https://github.com/arindas/laminarmq/tree/examples).

## Media

Media associated with the `laminarmq` project.

- `[BLOG]` [Building Segmented Logs in Rust: From Theory to Production!](https://arindas.github.io/blog/segmented-log-rust/)

## Design

This section describes the internal design of `laminarmq`.
Expand Down Expand Up @@ -640,47 +646,47 @@ All benchmarks in the reports have been run on a machine (HP Pavilion x360 Conve

### Selected Benchmark Reports

**Note**: We use the following names for different record sizes:

<table>
<tr>
<td><b>size_name</b></td>
<td><b>size</b></td>
<td><b>comments</b></td>
</tr>
<tr>
<td><code>tiny</code></td>
<td><code>12 bytes</code></td>
<td>none</td>
</tr>
<tr>
<td><code>tweet</code></td>
<td><code>140 bytes</code></td>
<td>none</td>
</tr>
<tr>
<td><code>half_k</code></td>
<td><code>560 bytes</code></td>
<td><code>≈ 512 bytes</code></td>
</tr>
<tr>
<td><code>k</code></td>
<td><code>1120 bytes</code></td>
<td><code>≈ 1024 bytes (1 KiB)</code></td>
</tr>
<tr>
<td><code>linked_in_post</code></td>
<td><code>2940 bytes</code></td>
<td><code>≤ 3000 bytes (3 KB)</code></td>
</tr>
<tr>
<td><code>blog</code></td>
<td><code>11760 bytes (11.76 KB)</code></td>
<td><code>4X linked_in_post</code></td>
</tr>
</table>

This section presents some selected benchmark reports:
This section presents some selected benchmark reports.

> **Note**: We use the following names for different record sizes:
>
> <table>
> <tr>
> <td><b>size_name</b></td>
> <td><b>size</b></td>
> <td><b>comments</b></td>
> </tr>
> <tr>
> <td><code>tiny</code></td>
> <td><code>12 bytes</code></td>
> <td>none</td>
> </tr>
> <tr>
> <td><code>tweet</code></td>
> <td><code>140 bytes</code></td>
> <td>none</td>
> </tr>
> <tr>
> <td><code>half_k</code></td>
> <td><code>560 bytes</code></td>
> <td><code>≈ 512 bytes</code></td>
> </tr>
> <tr>
> <td><code>k</code></td>
> <td><code>1120 bytes</code></td>
> <td><code>≈ 1024 bytes (1 KiB)</code></td>
> </tr>
> <tr>
> <td><code>linked_in_post</code></td>
> <td><code>2940 bytes</code></td>
> <td><code>≤ 3000 bytes (3 KB)</code></td>
> </tr>
> <tr>
> <td><code>blog</code></td>
> <td><code>11760 bytes (11.76 KB)</code></td>
> <td><code>4X linked_in_post</code></td>
> </tr>
> </table>
#### `commit_log` write benchmark with 1KB messages

Expand Down
26 changes: 24 additions & 2 deletions src/storage/commit_log/segmented_log/index.rs
View file
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
//! Provides components necessary for mapping record indices to store-file positions in segments.
//!
//! This module is used by the `segment` implementation to store mappings from record-indices to
//! positions on the `segment-store` file.
//! This module provides [`Index`] which maintains a mapping from logical record indices to
//! positions on the [`Store`](super::store::Store) of a [`Segment`](super::Segment).
use super::{
super::super::{AsyncConsume, AsyncIndexedRead, AsyncTruncate, Sizable, Storage},
Expand Down Expand Up @@ -51,6 +51,7 @@ pub struct IndexBaseMarker {
}

impl IndexBaseMarker {
/// Creates a new [`IndexBaseMarker`] with the given `base_index`.
pub fn new(base_index: u64) -> Self {
Self {
base_index,
Expand Down Expand Up @@ -149,6 +150,7 @@ impl SizedRecord for IndexBaseMarker {
}
}

/// Error type associated with operations on [`Index`].
#[derive(Debug)]
pub enum IndexError<StorageError> {
StorageError(StorageError),
Expand Down Expand Up @@ -199,6 +201,7 @@ macro_rules! idx_as_u64 {
}

impl IndexRecord {
/// Creates a new [`IndexRecord`] with the given position and [`RecordHeader`].
pub fn with_position_and_record_header<P: ToPrimitive>(
position: P,
record_header: RecordHeader,
Expand All @@ -225,6 +228,17 @@ impl IndexRecord {
/// <b>Fig:</b> <code>Segment</code> diagram showing <code>Index</code>, mapping logical indices
/// to<code>Store</code> positions.
/// </p>
///
/// [`Index`] also stores checksum and length information for every record with [`RecordHeader`].
/// This information is used to detect any data corruption on the underlying persistent media when
/// reading from [`Store`](super::store::Store).
///
/// ### Type parameters
/// - `S`
/// Underlying [`Storage`] implementation for storing [`IndexRecord`] instances
/// - `Idx`
/// Type to use for representing logical indices. (Usually an unsigned integer like u32, u64
/// usize, etc.)
pub struct Index<S, Idx> {
index_records: Option<Vec<IndexRecord>>,
base_index: Idx,
Expand All @@ -233,10 +247,12 @@ pub struct Index<S, Idx> {
}

impl<S, Idx> Index<S, Idx> {
/// Maps this [`Index`] to the underlying [`Storage`] implementation instance.
pub fn into_storage(self) -> S {
self.storage
}

/// Obtains the logical index of the first record in this [`Index`].
pub fn base_index(&self) -> &Idx {
&self.base_index
}
Expand All @@ -247,6 +263,10 @@ where
S: Storage,
Idx: Unsigned + FromPrimitive + Copy + Eq,
{
/// Returns the estimated number of [`IndexRecord`] instances stored in the given [`Storage`].
///
/// This function calculates this estimate by using the [`Storage`] size and the size of a
/// single [`IndexRecord`].
pub fn estimated_index_records_len_in_storage(
storage: &S,
) -> Result<usize, IndexError<S::Error>> {
Expand All @@ -261,6 +281,8 @@ where
Ok(estimated_index_records_len)
}

/// Reads and returns the `base_index` of the [`Index`] persisted on the provided [`Storage`]
/// instance by reading the [`IndexBaseMarker`] at [`INDEX_BASE_POSITION`].
pub async fn base_index_from_storage(storage: &S) -> Result<Idx, IndexError<S::Error>> {
let index_base_marker =
PersistentSizedRecord::<IndexBaseMarker, INDEX_BASE_MARKER_LENGTH>::read_at(
Expand Down

0 comments on commit b1de935

Please sign in to comment.