Add inline GEMM optimizations and general performance improvements by jfsantos · Pull Request #226 · sdatkinson/NeuralAmpModelerCore

jfsantos · 2026-02-12T23:52:28Z

Hand-optimized GEMM kernels for small matrices common in NAM models, gated by #ifdef NAM_USE_INLINE_GEMM with Eigen fallback. Includes:

Specialized Conv1D kernels: fused 4x4 and 2x2 kernel_size=3, plus fully-unrolled paths for 2x2 through 8x8 channel configurations
Conv1x1 inline specializations for all common size combinations
FiLM inline path with 4-element loop unrolling
GatingActivation/BlendingActivation inline paths
Branchless hardswish, 4-element loop unrolling for all activations
SiLU added to LUT enable/disable
Ring buffer refactored to Eigen block operations
memcpy replacements for pure copy operations in wavenet
Optimized single-channel output path in WaveNet::process
Buffer size benchmark tool (benchmodel_bufsize)

Developed with support and sponsorship from TONE3000

Hand-optimized GEMM kernels for small matrices common in NAM models, gated by #ifdef NAM_USE_INLINE_GEMM with Eigen fallback. Includes: - Specialized Conv1D kernels: fused 4x4 and 2x2 kernel_size=3, plus fully-unrolled paths for 2x2 through 8x8 channel configurations - Conv1x1 inline specializations for all common size combinations - FiLM inline path with 4-element loop unrolling - GatingActivation/BlendingActivation inline paths - Branchless hardswish, 4-element loop unrolling for all activations - SiLU added to LUT enable/disable - Ring buffer refactored to Eigen block operations - memcpy replacements for pure copy operations in wavenet - Optimized single-channel output path in WaveNet::process - Buffer size benchmark tool (benchmodel_bufsize) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

sdatkinson

[Haven't finished reviewing conv1d, film, gating_activations, wavenet.cpp, and benchmodel]

One crit on comments funny business.
Another crit: Can you add tests to ensure that the code is correct?
Other nits.

NAM/activations.cpp

sdatkinson · 2026-02-14T01:12:41Z

NAM/activations.h

+  // hardswish(x) = x * relu6(x + 3) / 6
+  //              = x * clamp(x + 3, 0, 6) / 6
+  const float t = x + 3.0f;
+  const float clamped = t < 0.0f ? 0.0f : (t > 6.0f ? 6.0f : t);


Interesting if this really is better? I'd be surprised if a compiler wouldn't figure out that these are the same.

I was surprised by this too, but it does make a difference. I can share the microbenchmark if you are interested in having a look.

nah I trust ya. sgtm

sdatkinson · 2026-02-14T01:13:34Z

NAM/activations.h

+    for (; pos + 3 < size; pos += 4)
+    {
+      // Branchless ReLU using conditional
+      const float v0 = data[pos], v1 = data[pos + 1];


Very interesting...I assume that this only works better on specific chips? No way some (most?) compilers don't know to do this?

sdatkinson · 2026-02-14T01:16:29Z

NAM/activations.h

+    // Process 4 elements at a time: swish(x) = x * sigmoid(x) = x / (1 + exp(-x))
+    for (; pos + 3 < size; pos += 4)
+    {
+      const float x0 = data[pos], x1 = data[pos + 1];


Nit: occurs to me looking at this versus the swish(data[pos]) on the left that some inlined "swish4" could look a tad cleaner? Not sure though.

sdatkinson · 2026-02-14T01:28:11Z

NAM/dsp.cpp

+    float* __restrict__ output_ptr = _output.data();
+    const float* __restrict__ bias_ptr = this->_bias.data();
+
+    // Specialized paths for common small channel counts


Nit: is it worth doing everything from 1 to 8?

sdatkinson · 2026-02-14T01:29:46Z

NAM/ring_buffer.cpp

+  // Write the input data at the write position using Eigen block operations
+  // This is more efficient than element-by-element copy as it allows
+  // the compiler to vectorize the operation.
+  _storage.middleCols(_write_pos, num_frames).noalias() = input.leftCols(num_frames);


I wonder why I did that...is there a chance that this isn't real-time safe? I need to check the tests...

NAM/wavenet.h

sdatkinson · 2026-02-14T01:31:51Z

tools/benchmodel_bufsize.cpp

+  }
+
+  // Turn on fast tanh approximation
+  nam::activations::Activation::enable_fast_tanh();


Nit: What do you think about making this an option?

I just got annoyed with it the other day independently of this PR :)

Yeah, let's add that as a flag.

sdatkinson · 2026-02-14T01:33:51Z

tools/benchmodel_bufsize.cpp

@@ -0,0 +1,96 @@
+#include <iostream>


Can you tell me (docstring?) the difference between this and benchmodel.cpp?

Sure, will do.

…ise ops ARM assembly analysis (-O2 -DNDEBUG) confirmed: - GCC auto-unrolls simple activation loops; manual 4-wide gives no benefit - expf() serializes sigmoid/SiLU; unrolling can't help - Eigen element-wise ops (.leftCols + .leftCols) produce identical codegen to raw float* loops when assertions are disabled Simplify 5 activation classes to use inline helpers (relu, sigmoid, etc.) and revert 3 wavenet element-wise operations back to Eigen expressions. Inline GEMM (Conv1x1/Conv1D), depthwise unrolling, FiLM unrolling, bias broadcast, and memcpy optimizations are retained — those show measurable wins on both desktop and Cortex-M7. Also restored comments that were accidentally removed from wavenet.h.

sdatkinson

There's a potential huge risk with non-contiguous matrices (like when we're doing gated/blended activations).

Can you tell me if there are any "snapshot" tests that would verify that the calculations with and without this flag are the same? I'm just really concerned about something being different that I missed.

[No changes required if you can verify that the things I was concerned about are correct. Sorry; it's just too much for me to fit in my head all at once and some "simple" proof would be a big help.]

sdatkinson · 2026-02-18T00:05:09Z

NAM/activations.h

+  // hardswish(x) = x * relu6(x + 3) / 6
+  //              = x * clamp(x + 3, 0, 6) / 6
+  const float t = x + 3.0f;
+  const float clamped = t < 0.0f ? 0.0f : (t > 6.0f ? 6.0f : t);


nah I trust ya. sgtm

sdatkinson · 2026-02-18T00:08:59Z

NAM/film.h

+    const int input_dim = (int)get_input_dim();
+    const float* __restrict__ input_ptr = input.data();
+    const float* __restrict__ scale_shift_ptr = scale_shift.data();
+    float* __restrict__ output_ptr = _output.data();


[C]

Oh, heck I just realized a potential problem with this:

What if the matrices aren't contiguous in memory?

This could happen e.g. with .topRows() since Eigen is column-major (or .leftCols() with a row-major matrix).

Is there a way to guarantee it doesn't happen? This was the problem with gating activations being wrong for a while. Is there a way we make sure that this doesn't happen?

sdatkinson · 2026-02-18T00:11:14Z

NAM/film.h

+      // scale = top input_dim rows, shift = bottom input_dim rows
+      for (int f = 0; f < num_frames; f++)
+      {
+        const float* __restrict__ in_col = input_ptr + f * input_rows;


Here's where you might land somewhere unexpected on f=1 on such a non-contiguous matrix.

sdatkinson · 2026-02-18T00:13:18Z

NAM/gating_activations.h

    // Validate input dimensions (assert for real-time performance)
-    const int total_channels = 2 * num_channels;
-    assert(input.rows() == total_channels);
+    assert(input.rows() == 2 * num_channels);


Nit: I'd changed my mind on these in favor of a throw that can be compiled out with #define NDEBUG

sdatkinson · 2026-02-18T00:20:05Z

NAM/wavenet.cpp

    // Use the GatingActivation class
    // Extract the blocks first to avoid temporary reference issues
    auto input_block = this->_z.leftCols(num_frames);
    auto output_block = this->_z.topRows(bottleneck).leftCols(num_frames);


cf. the non-contiguous concern.

Either this does an allocation, which means this shouldn't be real-time safe...

...but I haven't seen those tests fail, so that must mean that they address the memory as it's stored, and the activation concern I said must be actually an issue?

I really need to get to verifying that the results match the PyTorch...

sdatkinson · 2026-02-18T00:21:10Z

tools/benchmodel.cpp

-    const char* modelPath = argv[1];
-
-    std::cout << "Loading model " << modelPath << "\n";
+    std::cerr << "Usage: benchmodel <model_path> [--no-fast-tanh]\n";


sdatkinson · 2026-02-18T00:23:00Z

tools/benchmodel.cpp

-    size_t bufferSize = AUDIO_BUFFER_SIZE;
-    model->Reset(model->GetExpectedSampleRate(), bufferSize);
-    size_t numBuffers = (48000 / bufferSize) * 2;
+  if (model == nullptr)


Why'd the change detector go nuts on this file? Is it a formatting thing?

João Felipe Santos and others added 3 commits February 6, 2026 09:45

Fixed minor bug when building without inline GEMM

6c0942b

Added NAM_INLINE_GEMM build/test task to CI

5d9ed6c

sdatkinson requested changes Feb 14, 2026

View reviewed changes

jfsantos force-pushed the feature/inline-gemm branch from 704f309 to 7844a41 Compare February 16, 2026 18:44

Added option to disable fasttanh to benchmodel and benchmodel_bufsize

75db6b9

sdatkinson requested changes Feb 18, 2026

View reviewed changes

Conversation

jfsantos commented Feb 12, 2026

Uh oh!

sdatkinson left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sdatkinson left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants