[WIP] Rust Acceleration Kernel POC

[siddharthteotia]

Summary

Initially I was thinking of speeding up filters but filtering is generally not where most amount of time is spent especially when we have right indexes. Our indexes have proven to be efficient.

Full scans on range queries have proven to be expensive but that's also because we have barely scratched surface with SOTA (state of the art) range index in OLAP.

So the scope I am POCing is an end to end kernel written in Rust (glued via JNI) for GROUP BY and Aggregations with full benefits of vectorization + SIMD + cache aware hash tables formats.

Reasoning -- To reduce cost / QPS (read path specifically), we need to target bottlenecks on the query path -- whether it's within the operator itself, memory management, multi-threading or IO. GROUP BY Aggregations (commonly used by LinkedIn users of Pinot) is a frequent query that typically dominates workload cost. Use of Native opens a lot of opportunities for OLAP (e.g HW acceleration given repeatable nature of operations) in addition to the usual benefits (e.g memory safety) of a native language like Rust. Filtering is also a great candidate for acceleration to target but it's not inefficient for vast majority of workloads. Other candidates are memory mapped IO, column/segment readers, explicit core management etc.

Testing

Initial numbers observed after writing few kernels is upto 6x speed up (Rust + SIMD + JNI) and upto 2.5x speed up (Rust + JNI)

Design

This draft tracks ongoing progress. Strategic and detailed designs live in the diff. I will be converting this into a design doc on google doc to also facilitate reviews. Initially it's in MD file as I was trying to get POC started.

• RUST_REWRITE_DESIGN.md — strategic doc
• docs/native/phase-1-design.md — Phase 1 detailed design + measured Phase 1.A results (§11.A)

Status

• Phase 1.A complete. End-to-end vertical slice for SUM(LONG):
• New pinot-native module: Cargo workspace (ffi + kernels) + JNI bindings + Java loader
• Explicit SIMD kernel with runtime ISA dispatch: NEON / AVX2 / AVX-512DQ / scalar fallback
• Integration via `AggregationFunctionFactory
• Opt-in via pinot.native.aggregation.enabled (off by default)

[siddharthteotia]

Update ...

Have implemented 3 kernels and have taken observations with Rust Scalar + JNI and Rust + SIMD (Neon on ARM for now) + JNI.

Some interesting observations

x86 (Intel or AMD) testing is pending.

Will be sharing the performance numbers shortly

Moving on to implementing vectorized group by with cache friendly hash table format (Swiss) that is type aware and can decide to leverage vector instructions for probing as and when possible + specialized optimizations for single and multi key group by.

[dinoocch]

Neat! Just as a datapoint you could consider how much the manual simd gains over an auto-vectorizing friendly version: https://godbolt.org/z/aYMbnffxG

[siddharthteotia]

Right. I am measuring this specifically in the beginning itself.

If it's marginal on top of what Rust is already giving, then probably not worth the effort / complexity and we instead just focus on getting to Rust (while ensuring we are using Rust ergonomics with high craft).

[dinoocch]

Great, sounds good to me.

One thing that might be a bit less obvious is that since these are compiler directives #[cfg(target_arch = "aarch64")] will only do anything if the targeted cpu supports the options :)

Practically this means:
https://nnethercote.github.io/perf-book/build-configuration.html#cpu-specific-instructions

Wouldn't really recommend target-cpu=native for anything but exploring though (it means target exactly my cpu)

[dinoocch]

Okay really interesting.

Focusing just on neon -- the simd version is about twice as fast as scalar, due to accumulation + fp addition...

Benchmarking shows this is basically exactly the same as the simd version:

fn simple_sum(values: &[i64]) -> f64 {
      let mut acc = [0.0_f64; 8];
      let chunks = values.chunks_exact(8);
      let rem: f64 = chunks.remainder().iter().map(|&v| v as f64).sum();
      for c in chunks {
          for j in 0..8 {
              acc[j] += c[j] as f64;
          }
      }
      (acc[0]+acc[1]) + (acc[2]+acc[3]) + (acc[4]+acc[5]) + (acc[6]+acc[7])
}

AVX512: https://godbolt.org/z/aWxW3asG4
I tried to get godbolt to be happy with the neon targets but it's being a bit annoying...

---

Do we have formal rules about i64 overflowing sums? (I assume yes, which is why we are using f64 in the first place?), otherwise we would of course do:

pub fn sum_i64_to_f64(values: &[i64]) -> f64 {
  values.iter().copied().sum::<i64>() as f64
}

[dinoocch]

(totally understand and am on board w/ goals of getting something that we can see real results from -- just want to try and limit the amount of hand crafted simd / unsafe code that we need to maintain)

Auto-vectorization is also fragile but only in the sense that you may see performance differences

[dinoocch]

I might humbly suggest:

pub fn sum_i32_to_f64(values: &[i32]) -> f64 {
    values.iter().copied().map(|v| v as i64).sum::<i64>() as f64
}

Widening to i64 basically makes the overflow practically impossible.

This should also be pretty substantially faster than even the manually written simd version:
https://godbolt.org/z/5TzEPW5zP

[siddharthteotia]

Initial Performance Numbers on Neon. x86 numbers are pending

• Kernels POCd - SUM, MIN and MAX
• Distinct Count and GROUP BY is still being implemented along with e2e integration.

Summary table — native vs Java baseline speedup (10K block size)
  
  ┌─────┬────────┬───────┬────────┬────────┐
  │ Op  │  INT   │ LONG  │ FLOAT  │ DOUBLE │
  ├─────┼────────┼───────┼────────┼────────┤
  │ SUM │  5.92× │ 6.09× │  5.58× │  5.87× │
  ├─────┼────────┼───────┼────────┼────────┤
  │ MIN │ 11.83× │ 1.00× │ 11.53× │  6.52× │
  ├─────┼────────┼───────┼────────┼────────┤
  │ MAX │ 11.89× │ 0.99× │ 11.72× │  6.67× │
  └─────┴────────┴───────┴────────┴────────┘

Read further to understand the details

SUM Kernel speedup is uniform across types (5.6–6.1×)

• Always widens input → f64 inside the SIMD register and runs the same shape: convert → accumulate.
• Every type ends up at 2 f64 lanes/iter × 4-way ILP unroll = 8 elements/iter.
• The small variation comes from how much the conversion costs per type but Java's baseline scales the same way, so the net ratio stays narrow.

MIN/MAX swings widely (1.0× to 12×)

• INT/FLOAT MIN/MAX - The JVM path widens INT/FLOAT to f64 via getDoubleValuesSV() then loops Math.min/max(double, double).

  • The JIT does NOT autovectorize (confirmed via disassembly) that loop because the FP
    intrinsic needs NaN-aware semantics that don't lower to branchless SIMD.
  • OTOH - Native stays in source type and uses vminq_s32 / vminq_f32 (4 lanes / register). Java pays both the conversion cost AND the un-vectorized loop; native pays neither.

• DOUBLE MIN/MAX

  • SIMD lane count drops from 4 (f32) → 2 (f64), so the SIMD multiplier roughly halves.
  • Net speedup is ~half of FLOAT.

• LONG MIN/MAX

  • Math.min(long, long) IS auto-vectorizable by the JIT so the Java baseline is already at SIMD speed.
  • Native faces a second disadvantage: NEON has no native vminq_s64 — we synthesize it via vcgtq_s64 + vbslq_s64 CAS.
  • AVX-512F does have _mm512_min_epi64 natively. We expect a real LONG win on x86

Native Function

• NativeSumAggregationFunction.aggregate(....)

Current Java Baseline

• SumAggregationFunction.aggregate(...)

    ┌───────────────── Java layer ──────────────────┐   ┌──── JNI ────┐   ┌──────── Rust layer ────────┐

      NativeSumAggregationFunction                                        pinot_native_kernels::sum::
           .aggregate()                                                       long::sum_i64_to_f64
                │                                                                    ▲
                ▼                                                                    │
      BlockValSet.getLongValuesSV() ───► long[] ──► PinotNativeAgg ──► Java_..._sumLong (FFI)
                                                      .sumLong         ┌─pin via GetPrimitiveArrayCritical
                                                                        └─wrap in &[i64] slice

Currently the memory management between JVM -> JNI -> Rust is not yet optimized. Thought of an approach and noted it in the design doc but yet to work on it. Prioritizing getting end to end working and testing a real harness on x86 machine.

Used SIMD as well and measured it's benefit separately

• The _engine = native exercises the runtime-dispatched SIMD kernel.
• The _engine = native-scalar row exercises the same JNI path but invokes the 4-way unrolled scalar Rust kernel

Further verified through disassembly

• The dispatch function sum_i64_to_f64 checks is_aarch64_feature_detected!("neon") at runtime and routes to sum_i64_to_f64_neon

• vld1q_s64 (128-bit i64 load) → vcvtq_f64_s64 (native i64→f64 vector convert, scvtf.2d) → vaddq_f64 (128-bit f64 add), into four independent SIMD accumulators processing 8 elements per loop iteration. Final reduction uses pairwise vaddq_f64 then vaddvq_f64.

• Per-element throughput in the numbers below (~0.085 ns/elem ≈ 11.8 elem/ns at M-series clocks) is consistent only with full-vector NEON: the 4-way scalar fallback would land an order of magnitude higher.

• Equivalent SIMD bodies exist for AVX2 and AVX-512DQ on x86

JNI numbers - Want to measure this more but here is an initial glimpse

• JNI fixed cost is ≈ 85 ns per call — consistent across both native paths (since they share the dispatch code)
• The kernel function is dispatched at compile time via Rust generic monomorphization
• The time accounts for GetPrimitiveArrayCritical pinning, marshalling
• At _length = 100, JNI overhead shows up and Java wins by ~3.9× over native Rust (SIMD or scalar)
• At length = 10K, native Rust (with SIMD) is ~6.1× faster than Java; native Rust (scalar) is ~3.0× faster.
• At length = 100K the speedups are sustained. JNI overhead has fully amortized.

Native Rust Scalar vs Java scalar gap comes from explicit 4-way unrolling with four independent accumulators (good instruction level parallelism), no virtual-method. Java's hot SUM loop is JIT-friendly but accumulates into a single double, limiting ILP.

[dinoocch]

floats and doubles would guess are the places the simd wins due to NaN? Would have to spend more time thinking about the autovectorizing / how to structure it to make it obvious :(

[dinoocch]

Similarly, try:

#[inline(never)]
pub fn max_i32_to_f64(values: &[i32]) -> f64 {
    if values.is_empty() { return f64::NEG_INFINITY; }
    values.iter().copied().fold(i32::MIN, i32::max) as f64
}

https://godbolt.org/z/ofjG4n9z9

[dinoocch]

https://godbolt.org/z/Mfz1Ksx5f

pub fn max_i64_to_f64(values: &[i64]) -> f64 {
    if values.is_empty() { return f64::NEG_INFINITY; }
    values.iter().copied().fold(i64::MIN, i64::max) as f64
}

[dinoocch]

#[inline(never)]
pub fn min_i32_to_f64(values: &[i32]) -> f64 {
    if values.is_empty() { return f64::INFINITY; }
    values.iter().copied().fold(i32::MAX, i32::min) as f64
}

https://godbolt.org/z/7Ms6rc84f

[dinoocch]

https://godbolt.org/z/7KaqTjfMz

pub fn min_i64_to_f64(values: &[i64]) -> f64 {
    if values.is_empty() { return f64::INFINITY; }
    values.iter().copied().fold(i64::MAX, i64::min) as f64
}

[siddharthteotia]

@dinoocch - please excuse my Rust. I am also using this to get better at Rust so both hand-writing some code + AI based and eyeballing everything. This is why taking a bit longer. So idiomatically, it's not great yet. But will get there.

In any case, my goal is to not check-in this PR since reviewing 10K+ lines of Rust code in a single shot will be crazy. But please give feedback whatever you can. Once we get this to a point where data is convincing (may be another week or so), let's decide a path on how to break this down for shipping in a way that makes sense.

[dinoocch]

Yes, definitely understand. Just want to demonstrate that auto-vectorization in rust is very good / powerful so long as the compiler can make assumptions about the code :)

I have found that occasionally it seems like you need to "convince" the compiler that it's worth it by chunking...usually o3 will get the picture with some iterations.

Godbolt, especially with llvm-mca helps a ton

[siddharthteotia]

Makes sense. I will leverage the snippets you posted in the benchmark.

[siddharthteotia]

For GROUP BY, I am leveraging hashBrown (Rust port of Google's Swiss HashTable) This is same as DataFusion. Adding minor customizations.

Did some research / comparison with Clickhouse and DuckDB. They are using highly specialized stuff and multiple type specific flavors. Not taking that direction yet until get data points on end to end real GROUP BY workload.

[dinoocch]

For GROUP BY, I am leveraging hashBrown (Rust port of Google's Swiss HashTable) This is same as DataFusion. Adding minor customizations.

Did some research / comparison with Clickhouse and DuckDB. They are using highly specialized stuff and multiple type specific flavors. Not taking that direction yet until get data points on end to end real GROUP BY workload.

How does the memory management work with this + jni? Does rust own/allocate the memory and exposes functions to java? What is controlling the lifetime? I'm totally ignorant for how most jni works (ffm makes a bit more sense with the memory arenas?)?

[siddharthteotia]

Learning Notes for posterity - Java implementation's double whammy for MIN/MAX for INT, FLOAT and DOUBLE

Point 1 - Widening conversion to f64

MinAggregationFunction.aggregateGroupBySV calls blockValSet.getDoubleValuesSV() regardless of the column's storage type. The holder is always double. So for an INT column:

  double[] valueArray = blockValSet.getDoubleValuesSV();  // ← widens i32 → f64 for every row
  for (int i = 0; i < length; i++) {
      double cur = holder.getDouble(groupKey);
      holder.setDouble(groupKey, Math.min(cur, valueArray[i]));
  }

The getDoubleValuesSV() call materializes a double[] from the underlying int[] storage. For 100K rows, that's 100K conversion instructions. Exact setup analogous for FLOAT

Native Code skips this entirely. The kernel takes the raw int[] (or float[]) over JNI, stays in source type the whole loop, and only widens the single final result to f64 at the return boundary. We pay 1 conversion total, not N.

Point 2 - Math.min(double, double) blocks JIT auto- vectorization

Java's Math.min(double, double) library call is specified to:

• Propagate NaN: if either argument is NaN, return NaN
• Treat -0.0 < +0.0 strictly (most hardware min returns either)

These semantics don't match the raw hardware FP min instructions:

• x86 MINSD/MINPD: if either operand is NaN, returns the second operand. Asymmetric NaN handling which is wrong for Java.
• NEON vminq_f64: IS NaN-propagating per ARMv8 (ARMv8 follows IEEE 754-2019). Native uses this directly. But Java's JIT can't assume the host is ARMv8 — it has to emit code that works on x86 too.

So when the JIT sees this loop, it can't make a deterministic decision on what to do.

for (int i = 0; i < length; i++) {
      holder = Math.min(holder, valueArray[i]);
  }

It cannot legally lower it to a SIMD min instruction in one shot — the hardware min has different NaN behavior than Java spec. To vectorize correctly, the JIT would have to emit:

• SIMD min (gives wrong NaN behavior on x86)
• SIMD NaN-detect on inputs (extra compare + mask)
• SIMD blend: if NaN, return NaN; else return the min

That's 3 SIMD ops per element instead of 1. The JIT decides it's not worth the complexity and falls back to a scalar Math.min call per element.

Net result - Java's MIN() inner loop runs at ~1 element per cycle (scalar), not 2-4 per cycle.

---

Native Kernel handles this carefully:

• On NEON: vminq_f64 is natively NaN-propagating — exactly the semantics we want, zero overhead, just emit one instruction.
• On AVX2/AVX-512: the SIMD MINPD has the wrong NaN behavior, so we emit a sticky "saw NaN" accumulator (_mm*_cmp_pd(_CMP_UNORD_Q) OR'd into a mask) and inject NaN at the end if the mask is set. ~5-10%
  overhead vs the ideal NEON path, but still vastly faster than scalar.

[siddharthteotia]

Still digesting the hash tables and unsafe code.

@dinoocch - it's likely that latest push is very unclean. I was forced to push a lot from local on Monday (while I was iterating on it) as LinkedIn asked to re-image the laptop on Monday. So I backed up everything in rush

But yea feel free to leave comments... just want you to be aware ^^

[siddharthteotia]

Few next steps

• Multi column GROUP BY

  • Implemented the Rust Core end to end and tested it. See PackedKeyEncoder that packs dictionary IDs. Multiple types (fixed + var width) are supported. Based on Rayon, Morsel driven parallelism and work stealing.
  • At the combine level, wiring for multi key GROUP BY is remaining

• There is some marshalling that can be optimized away

  • I want to build DataTable directly from native arrays as opposed to feeding into JVM IndexedTable and forcing a complete rehash again which is not needed since I have implemented multi phase GROUP BY (and Top K) with radix partitioning + parallel merge.

  • At the segment level, I want to keep results as opaque native handle fed straight into combine -- no AggregationGroupByResult materialization / re-extract.

• I want to delete the custom hash table and just settle on std:hashmap (hashbrown) with ahash.

cc @dinoocch

[dinoocch]

flushing some inital thoughts on ffi/lib.rs, but only half done so far

[dinoocch]

Since we are working on bazelifying pinot stuff I wonder if we can pull directly up to jdk 25 for panama api stabilization. I wouldn't block on this though...

Definitely I think we should consider upgrading to jni 0.22 --
https://github.com/jni-rs/jni-rs/blob/master/crates/jni/docs/0.22-MIGRATION.md

Reading through it seems like they made a lot of substantive improvements...

[dinoocch]

0.21 -> 0.22 :)

[dinoocch]

We can simplify this macro by separating the jni logic from the naming...

This assumes you move to jni 0.22 which adds JPrimitiveArray::get_elements_critical +

#[inline(always)]
fn reduce_kernel<'local, T>(
    unowned: &mut EnvUnowned<'local>,
    array: &JPrimitiveArray<'local, T>,
    length: jint,
    empty: f64,
    kernel: impl Fn(&[T]) -> f64,
) -> jdouble
where
    T: TypeArray + Copy,
{
    if length <= 0 {
        return empty;
    }
    let outcome = unowned.with_env(|env| -> Result<f64, jni::errors::Error> {
        // SAFETY: ...
        let auto = unsafe { array.get_elements_critical(env, ReleaseMode::NoCopyBack) }?;
        let effective = (length as usize).min(auto.len());

        // SAFETY: ...
        let slice = unsafe { std::slice::from_raw_parts(auto.as_ptr() as *const T, effective) };
        Ok(kernel(slice))
    });
    match outcome.into_outcome() {
        Outcome::Ok(v) => v,
        Outcome::Err(_) | Outcome::Panic(_) => f64::NAN,
    }
}

macro_rules! define_reduce_jni {
    ($name:ident, $elem:ty, $kernel:path, $empty:expr) => {
        #[no_mangle]
        pub extern "system" fn $name<'local>(
            mut env: JNIEnv<'local>,
            _class: JClass<'local>,
            values: JPrimitiveArray<'local, $elem>,
            length: jint,
        ) -> jdouble {
            reduce_kernel(&mut env, &values, length, $empty, $kernel)
        }
    };
}

define_reduce_jni!(
    Java_org_apache_pinot_nativeengine_agg_PinotNativeAgg_sumLong,
    i64,
    sum::long::sum_i64_to_f64,
    0.0
);

The above also means you don't have to make sure that you match the java type and array types

[dinoocch]

I know we are getting rid of the handwritten hash table but, more ergonomically I think you can just do:

enum DriverBackend {
    Swiss(GroupByDriverDictInt<Table<i32>>),
    Hashbrown(GroupByDriverDictInt<HashbrownTable<i32>>),
}

struct BoxedDriver {
    backend: DriverBackend,
    dict_id_scratch: Vec<i32>,
    long_scratch: Vec<i64>,
    int_scratch: Vec<i32>,
    double_scratch: Vec<f64>,
    float_scratch: Vec<f32>,
}

Also as a small nit usually you wouldn't box the value for users / you would let the caller box it themselves.

It's also probably a minor performance note but this seems like it will have a decent amount of indirection -- each Vec is a pointer to some data (plus a length and capacity)

Also we could use a tiny macro to get rid of some of the match arms:

macro_rules! dispatch {
    (mut $self:expr, |$d:ident| $body:expr) => {
        match &mut $self.backend {
            DriverBackend::Swiss($d) => $body,
            DriverBackend::Hashbrown($d) => $body,
        }
    };
    ($self:expr, |$d:ident| $body:expr) => {
        match &$self.backend {
            DriverBackend::Swiss($d) => $body,
            DriverBackend::Hashbrown($d) => $body,
        }
    };
}

but this is being simplified so it doesn't matter :) just notes for learning

[dinoocch]

(nit) I'm kinda sure you don't need these casts?

[dinoocch]

ok need to come back here for usre.

[dinoocch]

uh same here, need to come back

[dinoocch]

Maybe we should use JPrimitiveArray::get_region?

pub fn get_region(
    &self,
    env: &Env<'_>,
    start: jsize,
    buf: &mut [T],
) -> Result<()>

This is kinda the signature you want all these to have, right? Plus it's safe

[dinoocch]

Hmm since we only need to hold one jni pin, would it make more sense to use get_elements_critical and operate on the slice without coping?

Edit: looks like the call tree below this wants an owned copy of the data which will live across jni calls (?) but I still need to study a bit better

[dinoocch]

We probably can avoid the pin here by using JPrimitiveArray::set_region

It probably has the same performance characteristics I would guess (and still copies memory) but should be cleaner at least?

[dinoocch]

Food for pondering -- it might be worth picking a specific quiet nan?

[dinoocch]

Consider https://crates.io/crates/ordered-float (we could benchmark to see the comparison)

I'm guessing it should be equivalent in performance (this has higher construction cost, that has higher Hash and Eq)

Overall I'm not too worried about perf with this since this would only be hit for non-dictionary floating point group-by right?

[dinoocch]

(nit) Usually you would not include trait bounds on the struct and keep them on only the impl

[dinoocch]

Todo, come back...

Things I'm thinking of:

• Is it worthwhile to have something like:

struct Partial {
  keys: Vec<K>,
  aggs: Vec<Option<AggState>>,
}

• I wonder if the Options are useful vs a zero value? (probably they are)

[dinoocch]

Instead of a tuple does it make sense to just use SegmentPartial

[dinoocch]

Do you need all of these bounds? Especially Copy is very restrictive?

[dinoocch]

(nit) you can do:

self.staging_aggs.fill_with(|| None);

[dinoocch]

Rayon is probably good for initial perf validations, it might be good to think about an executor abstraction or possibly consider async code (lots of async runtimes in rust can be very good task schedulers)

Probably we also need to at least limit the resource usage?

[dinoocch]

Tiny nit but I think you could instead do something like:

let first = columns[0];
let first_shift = self.shifts[0];
for i in 0..n {
    out[i] = (first[i] as u64 as i64) << first_shift;
}
for c in 1..columns.len() {
    let shift = self.shifts[c];
    let col = columns[c];
    for i in 0..n {
        out[i] |= (col[i] as u64 as i64) << shift;
    }
}

(to skip the memory fill to 0)

[dinoocch]

I wonder if it's better to do something like:

struct PackedField {
    width: u32,
    shift: u32,
    // maybe mask? this will remove the if from unpack...
}

struct PackedKeyEncoder {
    fields: Box<[PackedField]>,
}

[dinoocch]

I wonder if it's worthwhile to unroll common column lengths to avoid the multiple stores/loads and loop?

ie we might know that the majority of group-by's are less than 5.

[dinoocch]

Consider:

for (&id, &shift) in dict_ids.iter().zip(self.shifts.iter()) {
    packed |= (id as u64) << shift;
}

It's often a good idea to help convince the compiler that bounds checks are not needed

[dinoocch]

I suggest we remove this for now unless it's obviously better for performance

[dinoocch]

Also one contemplative thought -- if we pick a "standard" batch size of rows (or make it a compile time constant) a lot of these vecs can become fixed size arrays?

[dinoocch]

I wonder a bit if this scratch buffer actually helps us? Might be interesting to compare with a more naive implementation like:

pub fn process_block(&mut self, dict_ids: &[i32], values: &[i64]) {
    assert_eq!(dict_ids.len(), values.len());

    for (&dict_id, &value) in dict_ids.iter().zip(values) {
        let prev_len = self.table.len();
        let gid = self.table.probe_or_insert(dict_id) as usize;

        if gid == prev_len {
            self.keys.push(dict_id);
            self.sums.push(0);
        }

        self.sums[gid] = self.sums[gid].wrapping_add(value);
    }
}

[dinoocch]

I wonder a bit about the cost of shuffling string / byte objects across the JNI boundary :/ I guess this would involve at minimum memory copying overhead. Just food for thought

[dinoocch]

It's worth reviewing string interning implementations in the wild ~

https://crates.io/crates/lasso is especially interesting

[dinoocch]

Fun! Leaving some notes here for inspiration:

First, for storing small strings it might be worth checking out compact_str. It stores up to 24 byte strings on the stack and has some neat properties. And also some more specialized things like byteview

German strings go a bit further for long strings by encoding the prefix, check out: https://ltungv.com/note/an-optimization-thats-impossible-in-rust/