fix: compiled fused-training — standard-Adam default, OCP dispatch, MlpForward wiring, loud fallback

[ooples]

Closes #1447 (wire AIsEval fused primitives into NN layers + builder — framework side of AiDotNet.Tensors#436)
Closes #1470 (Transformer per-call training stalls — Noam LR frozen on the fused path)

Wires AiDotNet's NN training/inference onto AiDotNet.Tensors' fused-compiled kernels (the compile-once / replay-many path the Tensors micro-benchmarks beat PyTorch-CPU on), via open/closed self-describing interfaces — no central enum whitelist to keep in sync. Found by profiling the AIsEval PyTorch-vs-AiDotNet benchmark, where every Train() step silently fell back to the eager tape.

1. Default-optimizer gate (root cause of "compiled does nothing")

GetOrCreateBaseOptimizer defaulted UseAMSGrad = true (a non-standard band-aid from #1350); the fused mapper rejected AMSGrad, so every default-optimizer model fell back to the eager tape — silently. Reverted to standard Adam (matches PyTorch/TF/Optax, all default amsgrad=False).

2. Open/closed fused dispatch (replaces type-switch + enum whitelist)

• IFusedOptimizerSpec — optimizers self-describe their FusedOptimizerConfig; no central catalog.
• IFusedActivation — activations self-declare their FusedActivationType.
• TryGetFusedLrSchedule — LR schedulers map to the per-step fused LrSchedule (the fused kernel evaluates GetLr(step) every optimizer step, exactly like PyTorch fused=True).

3. #1470 — Adam+Noam on the fused fast path (true adaptive-LR fix)

Bumped AiDotNet.Tensors 0.86.6 → 0.88.0 for LrSchedule.Noam (Tensors #504). TryGetFusedLrSchedule now maps NoamSchedule → LrSchedule.Noam(d, warmup, factor) (replacing an eager-fallback workaround). The default Transformer recipe (Adam β₂=0.98 + Noam) now trains on the fused path with a correct per-step warmup ramp, bit-identical to the eager schedule. Verified: a default-Noam Transformer per-call Train engages the fused path 3200/3200 steps and converges (PPL 5.06, top-1 7/8) instead of freezing at the uniform floor.

4. Proper wiring — Predict → MlpForward

FeedForwardNeuralNetwork.Predict runs a pure dense+fused-activation stack as one IEngine.MlpForward call instead of the per-layer tape walk, via the activation interface. Falls back to generic Forward for anything unrepresentable.

5. Loud fallback (observability)

The fused path silently fell back at the default diagnostic level. Now emits a one-time warning per model naming the reason (suppressible via AIDOTNET_QUIET).

Coverage being completed on this branch

AiDotNet.Tensors 0.88.0 exposes 37 fused activations, 22 fused optimizers, 8 fused LR-schedule shapes. This PR expands AiDotNet's mappings toward full coverage, each gated by a numerical-parity test (fused result == eager result within tolerance) so no optimizer/activation is wired to a kernel whose math differs:

• Schedulers: Constant, Cosine, Exponential, Noam, + Step / Cyclic / OneCycle / LinearWarmupCosine.
• Optimizers: Adam, AdamW, SGD(+momentum), + the remaining fused kernels whose AiDotNet update math matches (AMSGrad, AdaMax, Nadam, RAdam, Adagrad, RMSprop, AdaDelta, Lion, …).
• Activations: ReLU/Sigmoid/Tanh/Identity, + the remaining exact-equivalent fused shapes (GELU, LeakyReLU, ELU, Softplus, Mish, Swish, HardSwish/HardSigmoid/HardTanh, ReLU6, SoftSign, CELU, …).
• Benchmarks: AIsEval PyTorch-parity harness re-run to confirm the compiled training plan beats PyTorch compiled (torch.compile) on the target shapes.

Builds clean on net10.0 + net471.

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features

  • Broad fused activation support and a fused MLP inference fast path.
  • Fused/compiled optimizer training paths added for many optimizers (Adam variants, SGD, RMSProp, Adagrad, AdaDelta, AdaMax, LAMB, Lion, Nadam).

• Improvements

  • Learning-rate schedules integrated with compiled training; improved fused-path diagnostics and fallback behavior.
  • Default Adam tuned for fused-kernel alignment.

• Documentation

  • Added PyTorch parity benchmark suite.

• Tests

  • New parity and integration tests validating fused inference and fused training.

• Dependencies

  • Updated native libraries to v0.90.2.

---

✅ Coverage completed + verified (parity-gated)

All wirings gated by a numerical-parity test; anything that diverged was left on eager and documented.

LR schedulers → LrSchedule: Constant, Cosine (fixed a pre-existing off-by-one), Exponential, Noam, Step, Cyclic(triangular). Step-for-step parity. (OneCycle deferred — AiDotNet uses linear warmup vs the kernel's cosine.)

Optimizers → fused kernels (fused-vs-eager training parity, Adam as control, all maxAbsDiff=0): SGD, Adam, AdamW, AMSGrad, AdaMax, Nadam, RMSprop, Adagrad, Lion, AdaDelta, LAMB. Expanded the stale 4-type allowlist → 20 and generalized the per-type fallback latch. (LARS/FTRL need params the config can't carry; RAdam/ASGD/Rprop have no AiDotNet class.)

Activations → FusedActivationType (parity ≤5e-7 via identity-weight FusedLinear): added Mish, SELU, Softplus, SoftSign, Sign, BentIdentity, Gaussian, LiSHT, SQRBF, ReLU6, HardSwish (on top of ReLU/Sigmoid/Tanh/Identity/GELU/LeakyReLU/SiLU/Swish). The gate caught HardSigmoid (0.2 slope vs kernel's x/6) — left on eager. Parametric (ELU/CELU/HardTanh/…) and RReLU/softmax-family deferred (documented).

Benchmark — vs torch.compile (TorchInductor), MLP CPU, steady-state

	AiDotNet fused	torch.compile	verdict
Training (compiled training plan)	~0.014 s/epoch	~0.084 s/epoch	AiDotNet ~6× faster ✅
Inference (Predict latency, bs32)	p95 1.27 ms	mean 0.32 ms	torch.compile ~4× faster

The compiled training plan beats torch.compile (~6×, and ~15× over AiDotNet eager). Inference-latency vs TorchInductor is an honest remaining gap. Harness: benchmarks/AiDotNet.PyTorchParity (now with --compile).

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

2 Skipped Deployments

Project	Deployment	Actions	Updated (UTC)
aidotnet_website	Ignored	Preview	May 31, 2026 5:42pm
aidotnet-playground-api	Ignored	Preview	May 31, 2026 5:42pm

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

Walkthrough

Adds IFusedActivation and IFusedOptimizerSpec contracts; implements TryGetFusedActivation across many activations (with parameter gating where needed); implements a fused dense+activation MLP inference fast-path; maps LR schedulers for fused optimizers and adds IFusedOptimizerSpec implementations for many optimizers; improves compiled-training fused gating and adds tests plus a PyTorch-parity benchmark project with Python scripts.

Changes

Fused Execution Infrastructure

Layer / File(s)	Summary
Fused activation contract & implementations src/ActivationFunctions/Fused/IFusedActivation.cs, src/ActivationFunctions/*Activation.cs	Adds public IFusedActivation.TryGetFusedActivation(out FusedActivationType) and implements it across many activations (identity→None, LeakyReLU alpha gating, SiLU→Swish, GELU documented numeric form; ELU/Mish intentionally non-claimed or documented).
Fused MLP inference fast-path src/NeuralNetworks/FeedForwardNeuralNetwork.cs	Predict enters inference mode, attempts TryFusedDensePredict which validates dense-only layers and scalar IFusedActivation mappings, collects weights/biases, calls AiDotNetEngine.MlpForward, and falls back to Forward on ineligibility or InvalidOperationException while restoring training-mode.
Fused optimizer contract src/Optimizers/Fused/IFusedOptimizerSpec.cs	Adds internal FusedOptimizerConfig and IFusedOptimizerSpec.TryGetFusedOptimizerConfig(out FusedOptimizerConfig) for fused-kernel optimizer configuration extraction.
Optimizers & LR schedule mapping src/Optimizers/*, src/Optimizers/GradientBasedOptimizerBase.cs	Multiple optimizers implement IFusedOptimizerSpec.TryGetFusedOptimizerConfig returning fused configs when adaptive LR is disabled and a fused LR schedule is available; TryGetFusedLrSchedule maps supported schedulers (null/constant, CosineAnnealing, Exponential, Noam, Step, symmetric Triangular cyclic) to fused LrSchedule.
NeuralNetworkBase: fused optimizer dispatch & fallback logging src/NeuralNetworks/NeuralNetworkBase.cs	TryMapToFusedOptimizerConfig dispatches via IFusedOptimizerSpec.TryGetFusedOptimizerConfig; adds a one-time fused-fallback TraceWarning guarded by _loggedFusedFallback; changes default base optimizer to Adam with UseAMSGrad=false.
Compiled training step: fused gating & latch src/Training/CompiledTapeTrainingStep.cs	Expands fused optimizer allowlist (includes AMSGrad); introduces per-thread _fusedUnavailableTypes to remember optimizer types that failed fused execution and skip retries; latch cleared on Invalidate().
Tests tests/*	Adds integration/unit tests validating fused optimizer engagement, fused activation numeric parity, LR schedule parity, LeakyReLU parameter gating, Mish/ELU non-claiming, and Predict stability on lazy initialization.

PyTorch-parity Benchmarks

Layer / File(s)	Summary
Solution & project files AiDotNet.sln, benchmarks/AiDotNet.PyTorchParity/*	Adds new benchmark project to solution, project file referencing src/AiDotNet.csproj, and .gitignore for benchmark artifacts.
C# benchmark harness & models benchmarks/AiDotNet.PyTorchParity/Program.cs	Adds harness that runs configured models (mlp, mlp-fused, cnn, lstm, transformer) for training and inference, measures timing, peak RSS, optional nvidia-smi, and writes indented JSON report.
PyTorch benchmark & compare scripts benchmarks/AiDotNet.PyTorchParity/pytorch/*	Adds Python parity benchmark, compare tool, requirements, and README describing parity workflow and measurement conventions.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related issues

• perf: Close per-model gaps surfaced by AIsEval fair-comparison vs PyTorch (LSTM, Transformer, MLP, regression-pipeline, OpenCL RSS) AiDotNet.Tensors#436 — Framework-side fused-kernel wiring and MLP/optimizer fusion objectives align with these changes.

Possibly related PRs

• ooples/AiDotNet#1386 — Related fused-Adam fused-step engagement tests and fused training path changes.
• feat: auto-compiled training integration (Phase 2) #1107 — Overlaps compiled training-step execution flow adjustments.
• feat: add learning rate scheduler integration and refactor diffusion architecture #574 — Related learning-rate scheduler and optimizer stepping integrations.

Suggested labels

feature, dependencies, testing, architecture, priority:p0

"A fused kernel hums beneath the code,
Activations mapped, the fast-path strode.
Adam learns while schedules climb,
Benchmarks hum and tests mark time.
Merge the harness — let metrics rhyme."

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch perf/fused-training-gate

[coderabbitai]

Actionable comments posted: 10

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

src/Training/CompiledTapeTrainingStep.cs (1)

351-353: ⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Update fused-optimizer constraints docs to include AMSGrad

The XML constraint text is now stale; code allows OptimizerType.AMSGrad (Line 408). Please update this block so behavior/docs stay aligned.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/Training/CompiledTapeTrainingStep.cs` around lines 351 - 353, The XML doc
for CompiledTapeTrainingStep currently lists only SGD, Adam, and AdamW as
supported by CompiledTrainingPlan{T}.ConfigureOptimizer but the code also allows
OptimizerType.AMSGrad; update the <item> text to include AMSGrad (and optionally
rephrase to "SGD, Adam, AdamW, and AMSGrad") and keep the note about using the
plain Step method or the eager tape path for other optimizer types so the
documentation matches the behavior of ConfigureOptimizer and references to Step
and the eager tape path remain accurate.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/ActivationFunctions/Fused/IFusedActivation.cs`:
- Around line 19-23: The IFusedActivation interface is an internal plumbing
contract and must not be public; change its declaration from public to internal
(i.e., make IFusedActivation internal) and ensure the related type
FusedActivationType is also internal or otherwise not exposed publicly so
callers outside the assembly cannot reference this plumbing; update any
implementing classes (names referencing IFusedActivation) to match the new
internal visibility and run compilation to fix any accessibility errors caused
by this change.

In `@src/ActivationFunctions/ReLUActivation.cs`:
- Around line 27-31: The ReLUActivation<T> class exposes the fused-kernel
plumbing member FusedActivationType as a public property; change it to an
explicit interface implementation so it is not part of the concrete public API.
Replace the public property declaration with an explicit implementation of
Fused.IFusedActivation.FusedActivationType (i.e. implement
AiDotNet.Tensors.Engines.FusedActivationType
Fused.IFusedActivation.FusedActivationType =>
AiDotNet.Tensors.Engines.FusedActivationType.ReLU) inside the ReLUActivation<T>
class so the member is only accessible via the Fused.IFusedActivation interface.

In `@src/ActivationFunctions/SigmoidActivation.cs`:
- Around line 36-40: The public FusedActivationType property on
SigmoidActivation<T> is exposing internal routing metadata; remove the public
auto-property and implement AiDotNet.Tensors.Engines.FusedActivationType as an
explicit interface member for Fused.IFusedActivation (i.e., implement
Fused.IFusedActivation.FusedActivationType =>
AiDotNet.Tensors.Engines.FusedActivationType.Sigmoid) inside the
SigmoidActivation<T> class so it is not part of the concrete public API but
still satisfies the interface contract.

In `@src/NeuralNetworks/NeuralNetworkBase.cs`:
- Around line 6666-6672: GetOrCreateBaseOptimizer relies on
AdamOptimizerOptions' constructor default for UseAMSGrad; make the non-AMSGrad
behavior explicit by creating the AdamOptimizerOptions instance, set its
UseAMSGrad property to false, and pass that options object into the
AdamOptimizer constructor (i.e., modify GetOrCreateBaseOptimizer to instantiate
a Models.Options.AdamOptimizerOptions<T, Tensor<T>, Tensor<T>> options variable,
set options.UseAMSGrad = false, then new AdamOptimizer(..., options)).
- Around line 5328-5341: The warning should not be emitted when compilation is
explicitly disabled; update the condition around the _loggedFusedFallback branch
to also check TensorCodecOptions.Current.EnableCompilation (or
TensorCodecOptions.EnableCompilation) and only log the fused-training fallback
when compilation is enabled and _mixedPrecisionContext is null; keep the
existing checks for _loggedFusedFallback, _mixedPrecisionContext, the
AIDOTNET_QUIET env var, and include _pendingFusedMissReason/GetType().Name in
the message as before so the alert only appears for unexpected fallbacks when
compilation was intended to run.

In `@src/Optimizers/AdamOptimizer.cs`:
- Around line 126-144: The public TryGetFusedOptimizerConfig method on
AdamOptimizer should be removed from the public API and implemented as an
explicit IFusedOptimizerSpec member; change the public method into an explicit
interface implementation of IFusedOptimizerSpec.TryGetFusedOptimizerConfig
within the AdamOptimizer class, keeping the existing logic (including the
UseAdaptiveLearningRate check, TryGetFusedLrSchedule call, and construction of
Fused.FusedOptimizerConfig using _options, GetCurrentLearningRate(),
TryGetFusedLrSchedule and the AMSGrad branch) so the behavior is identical but
the method is no longer part of the concrete public surface.

In `@src/Optimizers/AdamWOptimizer.cs`:
- Around line 146-162: The public method TryGetFusedOptimizerConfig on
AdamWOptimizer should be converted to an explicit interface implementation so
fused internals are not exposed on the concrete AdamWOptimizer API; locate the
TryGetFusedOptimizerConfig method (and its use of Fused.FusedOptimizerConfig,
TryGetFusedLrSchedule, GetCurrentLearningRate and _options) and change its
signature to implement the fused interface explicitly (e.g.
IFusedOptimizer.TryGetFusedOptimizerConfig) rather than a public member,
retaining the existing logic and return behavior so callers through the
interface still receive the Fused.FusedOptimizerConfig while the concrete
AdamWOptimizer class no longer exposes the fused plumbing publicly.

In `@src/Optimizers/Fused/IFusedOptimizerSpec.cs`:
- Around line 18-25: The Fused optimizer plumbing types are exposed publicly but
should be internal; change the accessibility of FusedOptimizerConfig (currently
declared as "public readonly record struct FusedOptimizerConfig(...)") to
internal (e.g., "internal readonly record struct FusedOptimizerConfig(...)") and
likewise change the other plumbing types mentioned around lines 51-58 to
internal; update any callers within the project (tests or other internal
classes) to use the now-internal types (no API consumers should be affected) and
ensure the file compiles after switching the access modifiers.

In `@src/Optimizers/StochasticGradientDescentOptimizer.cs`:
- Around line 70-81: The public TryGetFusedOptimizerConfig method on
StochasticGradientDescentOptimizer is exposing internal fused-config plumbing;
change it to an explicit interface implementation so it is not part of the
concrete public API. Locate the public bool TryGetFusedOptimizerConfig(out
Fused.FusedOptimizerConfig config) on class StochasticGradientDescentOptimizer
and convert it to an explicit implementation of the appropriate internal
interface (keep the logic that checks _options.UseAdaptiveLearningRate, calls
TryGetFusedLrSchedule, and uses GetCurrentLearningRate to build the
Fused.FusedOptimizerConfig), removing the public modifier so only the interface
exposes this member. Ensure the signature matches the internal interface exactly
and update accessibility accordingly.

In `@src/Training/CompiledTapeTrainingStep.cs`:
- Around line 397-408: The hot path currently admits
AiDotNet.Tensors.Engines.Compilation.OptimizerType.AMSGrad but relies on
exception-catching inside TryStepWithFusedOptimizer (and the per-step warning at
line ~640) to handle unsupported runtime builds, causing per-step exceptions and
log churn; modify the control flow so ConfigureOptimizer (or the plan selection
logic) performs a cheap runtime capability check for AMSGrad support (e.g.,
whether vMax buffer and FusedOptimizer.AMSGradUpdateSimd kernel are available)
and return a boolean/capability flag that the calling code
(CompiledTapeTrainingStep.TryStepWithFusedOptimizer) inspects before entering
the fused path, falling back once deterministically to the eager tape with a
single one-time warning instead of relying on exceptions; update usage of
optimizerType, ConfigureOptimizer, and TryStepWithFusedOptimizer to gate AMSGrad
by that capability flag.

---

Outside diff comments:
In `@src/Training/CompiledTapeTrainingStep.cs`:
- Around line 351-353: The XML doc for CompiledTapeTrainingStep currently lists
only SGD, Adam, and AdamW as supported by
CompiledTrainingPlan{T}.ConfigureOptimizer but the code also allows
OptimizerType.AMSGrad; update the <item> text to include AMSGrad (and optionally
rephrase to "SGD, Adam, AdamW, and AMSGrad") and keep the note about using the
plain Step method or the eager tape path for other optimizer types so the
documentation matches the behavior of ConfigureOptimizer and references to Step
and the eager tape path remain accurate.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: e67cea0d-07a3-4271-8787-dd0f2f62624a

📥 Commits

Reviewing files that changed from the base of the PR and between d05f43e and f8da771.

📒 Files selected for processing (14)

• src/ActivationFunctions/Fused/IFusedActivation.cs
• src/ActivationFunctions/GELUActivation.cs
• src/ActivationFunctions/IdentityActivation.cs
• src/ActivationFunctions/ReLUActivation.cs
• src/ActivationFunctions/SigmoidActivation.cs
• src/ActivationFunctions/TanhActivation.cs
• src/NeuralNetworks/FeedForwardNeuralNetwork.cs
• src/NeuralNetworks/NeuralNetworkBase.cs
• src/Optimizers/AdamOptimizer.cs
• src/Optimizers/AdamWOptimizer.cs
• src/Optimizers/Fused/IFusedOptimizerSpec.cs
• src/Optimizers/GradientBasedOptimizerBase.cs
• src/Optimizers/StochasticGradientDescentOptimizer.cs
• src/Training/CompiledTapeTrainingStep.cs

[coderabbitai]

Actionable comments posted: 4

♻️ Duplicate comments (1)

src/ActivationFunctions/Fused/IFusedActivation.cs (1)

28-35: ⚠️ Potential issue | 🔴 Critical | ⚖️ Poor tradeoff

BLOCKING: Fused activation dispatch contract must be internal, not public.

IFusedActivation is a fused-kernel routing interface consumed by internal layers (ActivationLayer.TryGetFusedActivationType, FeedForwardNeuralNetwork.TryFusedDensePredict). Exposing it as public violates the facade pattern—users should interact only with AiModelBuilder and configuration classes, not dispatch plumbing. All implementations use explicit interface syntax, so the interface visibility does not affect the concrete activation classes.

A prior review flagged this exact issue and marked it "Addressed in commits b029210 to a6851e4," yet the code remains public. Make it internal.

🔒 Proposed fix

-public interface IFusedActivation
+internal interface IFusedActivation
 {
     /// <summary>
     /// Reports the fused-kernel activation type equivalent to this activation, or
     /// returns <c>false</c> if this instance can't be reproduced by the kernel.
     /// </summary>
     bool TryGetFusedActivation(out FusedActivationType type);
 }

As per coding guidelines: "src/**: Users should ONLY interact with AiModelBuilder.cs and AiModelResult.cs" and "Prefer internal over public for plumbing/helper classes that users never instantiate or consume."

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/ActivationFunctions/Fused/IFusedActivation.cs` around lines 28 - 35,
Change the IFusedActivation interface from public to internal so the
fused-kernel routing contract is not exposed in the public API; update the
declaration of IFusedActivation accordingly (the explicit implementations on
concrete activation classes need no changes), and verify call sites like
ActivationLayer.TryGetFusedActivationType and
FeedForwardNeuralNetwork.TryFusedDensePredict still compile against the
now-internal interface.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@benchmarks/AiDotNet.PyTorchParity/pytorch/benchmark.py`:
- Line 276: The expression computing p95_idx uses an unnecessary int() cast
around round(); update the line that assigns p95_idx to remove int() so it reads
p95_idx = min(len(steady_sorted) - 1, round(0.95 * (len(steady_sorted) - 1))))
(referencing variable steady_sorted and the p95_idx assignment) ensuring the
result is still bounded by len(steady_sorted)-1; no other logic change required.
- Line 89: The __enter__ method's return type annotation uses unnecessary string
quotes; update the signature in the ResourceMonitor class by removing the quotes
so it reads a normal forward reference (i.e., change def __enter__(self) ->
"ResourceMonitor": to use ResourceMonitor without quotes), ensuring the
annotation matches the class name and Python's type hinting style.

In `@benchmarks/AiDotNet.PyTorchParity/pytorch/compare.py`:
- Around line 24-30: The helper _get lacks a return type annotation; update its
signature to include a return type and type for default (e.g. import Any from
typing and change to def _get(d: dict, *names: str, default: Any = None) ->
Any:) and ensure the typing import (from typing import Any) is added at the top;
this keeps behavior identical but provides explicit type information for callers
and linters.

In `@benchmarks/AiDotNet.PyTorchParity/pytorch/requirements.txt`:
- Line 3: Update the PyTorch minimum version in requirements.txt to avoid known
security vulnerabilities: replace the current "torch>=2.2" entry with a safer
minimum such as "torch>=2.5.0" or pin to a compatible patch series like
"torch~=2.6.0" so consumers use a known-safe release; ensure any CI or local
test docs that reference the "torch" requirement are updated accordingly.

---

Duplicate comments:
In `@src/ActivationFunctions/Fused/IFusedActivation.cs`:
- Around line 28-35: Change the IFusedActivation interface from public to
internal so the fused-kernel routing contract is not exposed in the public API;
update the declaration of IFusedActivation accordingly (the explicit
implementations on concrete activation classes need no changes), and verify call
sites like ActivationLayer.TryGetFusedActivationType and
FeedForwardNeuralNetwork.TryFusedDensePredict still compile against the
now-internal interface.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 5cd946b5-dc1b-4935-9648-9e103d21f2d0

📥 Commits

Reviewing files that changed from the base of the PR and between f8da771 and e32ade6.

📒 Files selected for processing (28)

• AiDotNet.sln
• benchmarks/AiDotNet.PyTorchParity/.gitignore
• benchmarks/AiDotNet.PyTorchParity/AiDotNet.PyTorchParity.csproj
• benchmarks/AiDotNet.PyTorchParity/Program.cs
• benchmarks/AiDotNet.PyTorchParity/README.md
• benchmarks/AiDotNet.PyTorchParity/pytorch/benchmark.py
• benchmarks/AiDotNet.PyTorchParity/pytorch/compare.py
• benchmarks/AiDotNet.PyTorchParity/pytorch/requirements.txt
• src/ActivationFunctions/ELUActivation.cs
• src/ActivationFunctions/Fused/IFusedActivation.cs
• src/ActivationFunctions/GELUActivation.cs
• src/ActivationFunctions/IdentityActivation.cs
• src/ActivationFunctions/LeakyReLUActivation.cs
• src/ActivationFunctions/MishActivation.cs
• src/ActivationFunctions/ReLUActivation.cs
• src/ActivationFunctions/SiLUActivation.cs
• src/ActivationFunctions/SigmoidActivation.cs
• src/ActivationFunctions/SwishActivation.cs
• src/ActivationFunctions/TanhActivation.cs
• src/NeuralNetworks/FeedForwardNeuralNetwork.cs
• src/NeuralNetworks/NeuralNetworkBase.cs
• src/Optimizers/AdamOptimizer.cs
• src/Optimizers/AdamWOptimizer.cs
• src/Optimizers/Fused/IFusedOptimizerSpec.cs
• src/Optimizers/StochasticGradientDescentOptimizer.cs
• src/Training/CompiledTapeTrainingStep.cs
• tests/AiDotNet.Tests/IntegrationTests/NeuralNetworks/FusedOptimizerIntegrationTests.cs
• tests/AiDotNet.Tests/UnitTests/NeuralNetworks/FusedInferenceParityTests.cs

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/Training/CompiledTapeTrainingStep.cs`:
- Line 128: The static field _fusedUnavailableTypes is being latched on any
exception which can permanently disable fused mode; update the logic so you only
add to _fusedUnavailableTypes for known non-transient exceptions (e.g.,
NotSupportedException/PlatformNotSupported or a specific
OptimizerUnsupportedException) instead of catching Exception, and ensure
Invalidate() actually clears or resets _fusedUnavailableTypes (or make it
instance-scoped) so transient failures don't permanently disable fused
execution; locate usages in CompiledTapeTrainingStep (the _fusedUnavailableTypes
field and the method where exceptions are caught and Invalidate() is
implemented) and change the catch to specific exception types and add a
safe-clear/reset in Invalidate().

In
`@tests/AiDotNet.Tests/IntegrationTests/Optimizers/FusedOptimizerParityTests.cs`:
- Around line 115-151: Each test (Adam_Control_FusedMatchesEager,
AdaMax_FusedMatchesEager_NoWorseThanAdam,
Nadam_FusedMatchesEager_NoWorseThanAdam) currently ignores the returned
trainDelta so they can pass with no parameter updates; update each test to
assert that trainDelta indicates actual parameter movement (e.g.,
Assert.True(trainDelta > 0 || maxAbs(trainDelta) > 1e-6) or similar non-zero
threshold) after calling Divergence(...) and include a clear failure message
mentioning the test name and that no training occurred; use the existing
trainDelta variable from the Divergence(...) call and keep the threshold
conservative (like 1e-6) so small-but-real updates pass while no-op runs fail.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 656f2a77-456b-4109-8222-49895364a143

📥 Commits

Reviewing files that changed from the base of the PR and between 53f58b4 and 7bba7ce.

📒 Files selected for processing (24)

• src/ActivationFunctions/BentIdentityActivation.cs
• src/ActivationFunctions/GaussianActivation.cs
• src/ActivationFunctions/HardSwishActivation.cs
• src/ActivationFunctions/LiSHTActivation.cs
• src/ActivationFunctions/MishActivation.cs
• src/ActivationFunctions/ReLU6Activation.cs
• src/ActivationFunctions/SELUActivation.cs
• src/ActivationFunctions/SQRBFActivation.cs
• src/ActivationFunctions/SignActivation.cs
• src/ActivationFunctions/SoftPlusActivation.cs
• src/ActivationFunctions/SoftSignActivation.cs
• src/LearningRateSchedulers/CyclicLRScheduler.cs
• src/Optimizers/AdaDeltaOptimizer.cs
• src/Optimizers/AdaMaxOptimizer.cs
• src/Optimizers/AdagradOptimizer.cs
• src/Optimizers/GradientBasedOptimizerBase.cs
• src/Optimizers/LAMBOptimizer.cs
• src/Optimizers/LionOptimizer.cs
• src/Optimizers/NadamOptimizer.cs
• src/Optimizers/RootMeanSquarePropagationOptimizer.cs
• src/Training/CompiledTapeTrainingStep.cs
• tests/AiDotNet.Tests/IntegrationTests/Optimizers/FusedActivationParityTests.cs
• tests/AiDotNet.Tests/IntegrationTests/Optimizers/FusedLrScheduleMappingTests.cs
• tests/AiDotNet.Tests/IntegrationTests/Optimizers/FusedOptimizerParityTests.cs