Disks keep waking up every 90 minutes - TrueNAS 25.10.1

[diego351]

TrueNAS Scale: 25.10.1
System Dataset Pool: boot-pool
latest spindown-fix.sh and v25 patch provided here.

Spindown time set to 5 minutes for all HDDs, power management set to 1.

Script triggered on boot:

Q: Should it really be Pre Init? I’m pretty sure i’ve seen it PostInit in previous fix revision in this repo.

RAIDZ pool named storage:

raidz1:
	3x SATA Seagate EXOS X24 24TB

special vdev:
	mirrored 2x SATA Intel SSD D3-S4610 960G

zil:
	1x NVME Intel Optane 800p 118G

l2arc:
	1x NVME Silicon Power P34A80 2T

Power consumption log during night, with running middleware service on:


and with middlewared service stopped:

Running hdd-wake-trace.sh on balanced profile logs wake-events.tsv:

1781145196	2026-06-11 04:33:16	partial disk wake	14478	/dev/sdd=ST24000NM000H-3KS103_ZYD0BVY4	/root/hdd-wake-trace-logs/2026-06-11_04-33-25_storage_partial-disk-wake_st24000nm000h-3ks103_zyd0bvy4.log
1781150600	2026-06-11 06:03:20	partial disk wake	5404	/dev/sdd=ST24000NM000H-3KS103_ZYD0BVY4	/root/hdd-wake-trace-logs/2026-06-11_06-03-25_storage_partial-disk-wake_st24000nm000h-3ks103_zyd0bvy4.log
1781155997	2026-06-11 07:33:17	partial disk wake	5397	/dev/sdd=ST24000NM000H-3KS103_ZYD0BVY4	/root/hdd-wake-trace-logs/2026-06-11_07-33-25_storage_partial-disk-wake_st24000nm000h-3ks103_zyd0bvy4.log
1781161396	2026-06-11 09:03:16	partial disk wake	5399	/dev/sdd=ST24000NM000H-3KS103_ZYD0BVY4	/root/hdd-wake-trace-logs/2026-06-11_09-03-25_storage_partial-disk-wake_st24000nm000h-3ks103_zyd0bvy4.log

Extra info:



cat spindown-fix.sh | grep SCRIPT_DIR/
PATCH="$SCRIPT_DIR/spindown-v25.patch"
BOOT_SCRIPT="$SCRIPT_DIR/spindown-overlay-mount.sh"

./spindown-fix.sh status

Overlay status:

File:
  /usr/lib/python3/dist-packages/middlewared/plugins/pool_/dataset_encryption_info.py
Overlay:
  present: /root/TrueNAS-Tricks/my-overlay/usr/lib/python3/dist-packages/middlewared/plugins/pool_/dataset_encryption_info.py
Mount:
/usr/lib/python3/dist-packages/middlewared/plugins/pool_/dataset_encryption_info.py boot-pool/ROOT/25.10.1/root[/TrueNAS-Tricks/my-overlay/usr/lib/python3/dist-packages/middlewared/plugins/pool_/dataset_encryption_info.py] zfs rw,nosuid,nodev,relatime,xattr,noacl,casesensitive

File:
  /usr/lib/python3/dist-packages/middlewared/utils/disks_/disk_class.py
Overlay:
  present: /root/TrueNAS-Tricks/my-overlay/usr/lib/python3/dist-packages/middlewared/utils/disks_/disk_class.py
Mount:
/usr/lib/python3/dist-packages/middlewared/utils/disks_/disk_class.py boot-pool/ROOT/25.10.1/root[/TrueNAS-Tricks/my-overlay/usr/lib/python3/dist-packages/middlewared/utils/disks_/disk_class.py] zfs rw,nosuid,nodev,relatime,xattr,noacl,casesensitive

File:
  /usr/lib/python3/dist-packages/middlewared/alert/source/smart.py
Overlay:
  present: /root/TrueNAS-Tricks/my-overlay/usr/lib/python3/dist-packages/middlewared/alert/source/smart.py
Mount:
/usr/lib/python3/dist-packages/middlewared/alert/source/smart.py boot-pool/ROOT/25.10.1/root[/TrueNAS-Tricks/my-overlay/usr/lib/python3/dist-packages/middlewared/alert/source/smart.py] zfs rw,nosuid,nodev,relatime,xattr,noacl,casesensitive

File:
  /usr/lib/python3/dist-packages/middlewared/plugins/disk.py
Overlay:
  present: /root/TrueNAS-Tricks/my-overlay/usr/lib/python3/dist-packages/middlewared/plugins/disk.py
Mount:
/usr/lib/python3/dist-packages/middlewared/plugins/disk.py boot-pool/ROOT/25.10.1/root[/TrueNAS-Tricks/my-overlay/usr/lib/python3/dist-packages/middlewared/plugins/disk.py] zfs rw,nosuid,nodev,relatime,xattr,noacl,casesensitive

So fixes are there, are properly mounted and in the right place.

running hdd-wake-trace.sh on balanced profile i see in event detail logs:

== Wake analysis summary ==
HIGH:    recurrence         Wake interval is approximately 90 minutes (5399s since previous event).
MEDIUM:  SMART              SMART or TrueNAS disk polling activity was seen around the wake. Check whether it appears before or after wake_epoch.
MEDIUM:  ZFS                ZFS kernel worker or txg_sync I/O appeared immediately after wake.
MEDIUM:  ZFS                ZFS workers were active around the wake. This shows the executor, but not necessarily the original caller.

== Human-readable interpretation ==
- The wake cadence is highly regular, which points toward a timer, polling cycle, middleware task, or scheduled client behaviour.
- ZFS workers were active. This usually means ZFS executed I/O, but it may not be the original initiator.
- SMART or disk polling activity was seen. Check whether it appears before or after the wake timestamp.
- The detection time is not the exact spin-up instant; the real trigger may be inside the previous poll interval.


== SMART / disk polling context ==
-- SMART processes currently visible --
 278278    1130 root     Thu Jun 11 09:03:36 2026 smartctl        smartctl -x /dev/sdb -jc

which makes me think that patch isn’t working as pre-patch that line in /usr/lib/python3/dist-packages/middlewared/utils/disks_/disk_class.py file looked like this:


cmd = ["smartctl", "-x", self.devpath]

but for me it seems it’s correctly patched:

cat /usr/lib/python3/dist-packages/middlewared/utils/disks_/disk_class.py | grep smartctl
    def __run_smartctl_cmd_impl(self, cmd: list[str], raise_alert: bool = True) -> str:
    def smartctl_info(self, return_json: bool = False, raise_alert: bool = True) -> dict | str:
        """Return smartctl -x information.
        cmd = ["smartctl", "-n", "standby", "-x"]
        stdout = self.__run_smartctl_cmd_impl(cmd, raise_alert)
    def smartctl_test(
        self.__run_smartctl_cmd_impl(["smartctl", self.devpath, "-t", ttype], raise_alert)

Any ideas? Should it really be launched in Pre Init? I'm happy to provide more info if needed. Thanks.

Full logs:

2026-06-11_04-33-25_storage_partial-disk-wake_st24000nm000h-3ks103_zyd0bvy4.log
2026-06-11_06-03-25_storage_partial-disk-wake_st24000nm000h-3ks103_zyd0bvy4.log
2026-06-11_07-33-25_storage_partial-disk-wake_st24000nm000h-3ks103_zyd0bvy4.log
2026-06-11_09-03-25_storage_partial-disk-wake_st24000nm000h-3ks103_zyd0bvy4.log

[itiligent]

I went through a similar issue in testing and refining the patch script....

Every 90 mins is the default schedule for a bunch of smarctl things that the scripts is supposed to disable, which points us in a fairly clear direction. Do the drives ever reach standby?

During testing, in some cases the overlay gets in a race condition at boot (even as a pre-int command.).

The 90 minute schedule points to the original config files being parsed at boot BEFORE the overlay patch takes effect... so its created, , but not applied.

Try this:
Restart middleware manually after the boot is settled and see what happens. If that works, see my notes in the readme around making this middleware restart a short time after boot with ENABLE_DELAYED_RESTART=yes DELAY_SECONDS=300 bash /path/to/spindown-overlay-mount.sh

Failing that, have a deeper read of this to run my hdd-wake-trace script which will watch, log, and track sequences/patterns of wake events to help point you toward exactly what processes are touching the drives and when.

[mepreston]

https://ixsystems.atlassian.net/browse/NAS-137887