Rust 每日一库 on wsmの随记

[Rust 每日一库] base64

Tue, 26 Nov 2024 19:10:59 +0800

又功能一相当基础的小而美库

核心功能

base64 库把核心功能在 engine::Engine

pub trait Engine {
    fn internal_encode(&self, input: &[u8], output: &mut [u8]) -> usize;
    fn internal_decoded_len_estimate(&self, input_len: usize) -> Self::DecodeEstimate;
    fn internal_decode(
        &self,
        input: &[u8],
        output: &mut [u8],
        decode_estimate: Self::DecodeEstimate,
    ) -> Result<DecodeMetadata, DecodeSliceError>;

}

然后它实现了一个 GeneralPurpose 实现了这个 Trait

值得注意的是它的实现用到了循环展开

fn read_u64(s: &[u8]) -> u64 {
    u64::from_be_bytes(s[..8].try_into().unwrap())
}
fn internal_encode(&self, input: &[u8], output: &mut [u8]) -> usize {
    let mut input_index: usize = 0;

    const BLOCKS_PER_FAST_LOOP: usize = 4;
    const LOW_SIX_BITS: u64 = 0x3F;

    // we read 8 bytes at a time (u64) but only actually consume 6 of those bytes. Thus, we need
    // 2 trailing bytes to be available to read..
    let last_fast_index = input.len().saturating_sub(BLOCKS_PER_FAST_LOOP * 6 + 2);
    let mut output_index = 0;

    if last_fast_index > 0 {
        while input_index <= last_fast_index {
            // Major performance wins from letting the optimizer do the bounds check once, mostly
            // on the output side
            let input_chunk =
                &input[input_index..(input_index + (BLOCKS_PER_FAST_LOOP * 6 + 2))];
            let output_chunk =
                &mut output[output_index..(output_index + BLOCKS_PER_FAST_LOOP * 8)];

            // Hand-unrolling for 32 vs 16 or 8 bytes produces yields performance about equivalent
            // to unsafe pointer code on a Xeon E5-1650v3. 64 byte unrolling was slightly better for
            // large inputs but significantly worse for 50-byte input, unsurprisingly. I suspect
            // that it's a not uncommon use case to encode smallish chunks of data (e.g. a 64-byte
            // SHA-512 digest), so it would be nice if that fit in the unrolled loop at least once.
            // Plus, single-digit percentage performance differences might well be quite different
            // on different hardware.

            let input_u64 = read_u64(&input_chunk[0..]);

            output_chunk[0] = self.encode_table[((input_u64 >> 58) & LOW_SIX_BITS) as usize];
            output_chunk[1] = self.encode_table[((input_u64 >> 52) & LOW_SIX_BITS) as usize];
            output_chunk[2] = self.encode_table[((input_u64 >> 46) & LOW_SIX_BITS) as usize];
            output_chunk[3] = self.encode_table[((input_u64 >> 40) & LOW_SIX_BITS) as usize];
            output_chunk[4] = self.encode_table[((input_u64 >> 34) & LOW_SIX_BITS) as usize];
            output_chunk[5] = self.encode_table[((input_u64 >> 28) & LOW_SIX_BITS) as usize];
            output_chunk[6] = self.encode_table[((input_u64 >> 22) & LOW_SIX_BITS) as usize];
            output_chunk[7] = self.encode_table[((input_u64 >> 16) & LOW_SIX_BITS) as usize];

            let input_u64 = read_u64(&input_chunk[6..]);

            output_chunk[8] = self.encode_table[((input_u64 >> 58) & LOW_SIX_BITS) as usize];
            output_chunk[9] = self.encode_table[((input_u64 >> 52) & LOW_SIX_BITS) as usize];
            output_chunk[10] = self.encode_table[((input_u64 >> 46) & LOW_SIX_BITS) as usize];
            output_chunk[11] = self.encode_table[((input_u64 >> 40) & LOW_SIX_BITS) as usize];
            output_chunk[12] = self.encode_table[((input_u64 >> 34) & LOW_SIX_BITS) as usize];
            output_chunk[13] = self.encode_table[((input_u64 >> 28) & LOW_SIX_BITS) as usize];
            output_chunk[14] = self.encode_table[((input_u64 >> 22) & LOW_SIX_BITS) as usize];
            output_chunk[15] = self.encode_table[((input_u64 >> 16) & LOW_SIX_BITS) as usize];

            let input_u64 = read_u64(&input_chunk[12..]);

            output_chunk[16] = self.encode_table[((input_u64 >> 58) & LOW_SIX_BITS) as usize];
            output_chunk[17] = self.encode_table[((input_u64 >> 52) & LOW_SIX_BITS) as usize];
            output_chunk[18] = self.encode_table[((input_u64 >> 46) & LOW_SIX_BITS) as usize];
            output_chunk[19] = self.encode_table[((input_u64 >> 40) & LOW_SIX_BITS) as usize];
            output_chunk[20] = self.encode_table[((input_u64 >> 34) & LOW_SIX_BITS) as usize];
            output_chunk[21] = self.encode_table[((input_u64 >> 28) & LOW_SIX_BITS) as usize];
            output_chunk[22] = self.encode_table[((input_u64 >> 22) & LOW_SIX_BITS) as usize];
            output_chunk[23] = self.encode_table[((input_u64 >> 16) & LOW_SIX_BITS) as usize];

            let input_u64 = read_u64(&input_chunk[18..]);

            output_chunk[24] = self.encode_table[((input_u64 >> 58) & LOW_SIX_BITS) as usize];
            output_chunk[25] = self.encode_table[((input_u64 >> 52) & LOW_SIX_BITS) as usize];
            output_chunk[26] = self.encode_table[((input_u64 >> 46) & LOW_SIX_BITS) as usize];
            output_chunk[27] = self.encode_table[((input_u64 >> 40) & LOW_SIX_BITS) as usize];
            output_chunk[28] = self.encode_table[((input_u64 >> 34) & LOW_SIX_BITS) as usize];
            output_chunk[29] = self.encode_table[((input_u64 >> 28) & LOW_SIX_BITS) as usize];
            output_chunk[30] = self.encode_table[((input_u64 >> 22) & LOW_SIX_BITS) as usize];
            output_chunk[31] = self.encode_table[((input_u64 >> 16) & LOW_SIX_BITS) as usize];

            output_index += BLOCKS_PER_FAST_LOOP * 8;
            input_index += BLOCKS_PER_FAST_LOOP * 6;
        }
    }

    // Encode what's left after the fast loop.

    const LOW_SIX_BITS_U8: u8 = 0x3F;

    let rem = input.len() % 3;
    let start_of_rem = input.len() - rem;

    // start at the first index not handled by fast loop, which may be 0.

    while input_index < start_of_rem {
        let input_chunk = &input[input_index..(input_index + 3)];
        let output_chunk = &mut output[output_index..(output_index + 4)];

        output_chunk[0] = self.encode_table[(input_chunk[0] >> 2) as usize];
        output_chunk[1] = self.encode_table
            [((input_chunk[0] << 4 | input_chunk[1] >> 4) & LOW_SIX_BITS_U8) as usize];
        output_chunk[2] = self.encode_table
            [((input_chunk[1] << 2 | input_chunk[2] >> 6) & LOW_SIX_BITS_U8) as usize];
        output_chunk[3] = self.encode_table[(input_chunk[2] & LOW_SIX_BITS_U8) as usize];

        input_index += 3;
        output_index += 4;
    }

    if rem == 2 {
        output[output_index] = self.encode_table[(input[start_of_rem] >> 2) as usize];
        output[output_index + 1] =
            self.encode_table[((input[start_of_rem] << 4 | input[start_of_rem + 1] >> 4)
                & LOW_SIX_BITS_U8) as usize];
        output[output_index + 2] =
            self.encode_table[((input[start_of_rem + 1] << 2) & LOW_SIX_BITS_U8) as usize];
        output_index += 3;
    } else if rem == 1 {
        output[output_index] = self.encode_table[(input[start_of_rem] >> 2) as usize];
        output[output_index + 1] =
            self.encode_table[((input[start_of_rem] << 4) & LOW_SIX_BITS_U8) as usize];
        output_index += 2;
    }

    output_index
}

O3 优化下，x86_64 的汇编代码就是朴素的一堆 mov, and 和 shr；arm 汇编倒是做了乱序处理；然而实测在 x86_64 (ryzen 6800h) 上展开与否并不会显著影响性能（提升平均 5% 左右且不稳定）。个人不是很喜欢这个做法

[Rust 每日一库] bitflags

Wed, 23 Oct 2024 23:10:59 +0800

bitflags 在宏三大法器中抢夺出了前四名中的一席之地。它本身的功能相当简单。直接看代码：

bitflags::bitflags! {
    pub struct Flags: u32 {
        const A = 0b00000001;
        const B = 0b00000010;
        const C = 0b00000100;
    }
}

或是

pub struct Flags(u32);

bitflags! {
    impl Flags: u32 {
        const A = 0b00000001;
        const B = 0b00000010;
        const C = 0b00000100;
    }
}

然后就可以把 Flags 类似 u128 一样计算使用，不同的是它基于 Flags trait 实现了一些好方法，如同官方 slice 一样，让我们能够轻松使用。

这是一种值得学习的范式：用 declarative macro 实现最小化扩展，然后用一个 trait 实现更多方法。

宏

让我们深入看看 bitflags 的实际实现：教科书式的 declarative macro。

$(#[$outer:meta])*
$vis:vis struct $BitFlags:ident: $T:ty {
    $(
        $(#[$inner:ident $($args:tt)*])*
        const $Flag:tt = $value:expr;
    )*
}

$($t:tt)*

首先它处理了内外的 metadata；以支持诸如嵌套 macro、文档注释、编译标记等额外信息。

然后是逐步的函数式 declarative macro 实现。

它在最后处理了额外的 Token Tree，以支持多类型定义。

总之简单好用就完了。

[Rust 每日一库] hashbrown

Wed, 23 Oct 2024 19:52:59 +0800

hashbrown 库是 Rust 转写的 Google SwissTable，算法细节可以看 CppCon talk。它也是 1.36 之后版本的 Rust 标准库 HashTable 实现。

Layout

大体架构上，它实现了一个 RawTable<T>。不同于 CppCon 中提到的一个 ctrl 一个 bucket 两个向量，它的实现只用了一个 ctrl 指针。

源码中有以下说明：

                       `self.table.ctrl.cast()` returns pointer that
                       points here (to the end of `T0`)
                         ∨
[Pad], T_n, ..., T1, T0, |CT0, CT1, ..., CT_n|, CTa_0, CTa_1, ..., CTa_m
                          \________  ________/
                                   \/
      `n = buckets - 1`, i.e. `RawTable::buckets() - 1`

where: T0...T_n  - our stored data;
       CT0...CT_n - control bytes or metadata for `data`.
       CTa_0...CTa_m - additional control bytes, where `m = Group::WIDTH - 1` (so that the search
                       with loading `Group` bytes from the heap works properly, even if the result
                       of `h1(hash) & self.bucket_mask` is equal to `self.bucket_mask`). See also
                       `RawTableInner::set_ctrl` function.

P.S. `h1(hash) & self.bucket_mask` is the same as `hash as usize % self.buckets()` because the number
of buckets is a power of two, and `self.bucket_mask = self.buckets() - 1`.

本质上是 Grouped Vector，让人眼前一亮！这种做法感觉对缓存非常友好。

[Rust 每日一库] 引言

Tue, 22 Oct 2024 23:10:59 +0800

Rust 提供了 crates.io 作为官方在线库服务；不过即使是拥有 Rust 语言本身的高要求，crates.io 里已然有着和 npm registry 一样的毛病：不少库看似花哨，实际性能低下、包装过多、依赖繁重。

本系列计划或浅或深地按照 crates.io 上热门排名的顺序，选择一些库进行试用和学习；不爽的点可能会批判；但不会深入探究用法和功能，对应需求建议参考对应库的 examples/ 文件夹。