WIP - Huge performance upgrade for SSD based devices & cache rework for less iPod reads

[B0N3head]

More sync more faster

I had a look into slowdown with initial syncing (as it took an ungodly long time with 4k songs). I found that each song was 1 by 1 being checked with fpcalc.exe to ID tracks, which is fine, HOWEVER on a solid state based iPods (or any modded one iFlashQuad for example) this is barely using 1/4th of the USB bandwidth available. So now when you connect a device to iOpenPod it will ask if the connected device is SSD based. If it is then it will use 3-4 workers (instead of the single) to ID the music not only on your local PC library but also on your iPod.

I'm currently going through the rest of the codebase to try incorporate more speedups to do with single IO functions for SSD based speedups.

Changes made so far*

• Persisted per-device settings, device_tags.json keyed by serial
• Popup on first connect, _ensure_ipod_drive_tag() fires on connect + device change, skips if already set
• Device name and serial stored in saved tags set_ipod_hdd_tag now auto-extracts from device_info and persists alongside the bool (this is so later in settings users can change each iPods "hdd tag" without having to have the iPod plugged in or knowing the SN)
• Tag used across all workers, BackupCreate, BackupRestore, BackSync, SyncDiff (bootstrap_workers), SyncExecute (pre-sync backup + write concurrency)
• New settings page entry, "Device" section in General page. ComboRow always visible, enabled only when device connected, reads/writes device_tags.json directly, skips _save() cycle
• Backup/Sync pages now both have "Elapsed Time" and have their UI closer matched to each other (to use for "comparisons/benchmarks")

*originally built off 1.0.49 and updated to v1.0.51 so there might be minor differences between this fork and v1.0.51

Current coverage map

Operation	HDD	SSD
Pre-sync backup hashing	1 worker	4 workers
Manual backup hashing	1 worker	4 workers
Track copy / write to iPod	1 write slot	up to 4 write slots
Bootstrap fingerprinting	1 fpcalc	up to 3 fpcalcs
BackSync iPod fingerprinting	1 fpcalc	up to 3 fpcalcs

I'd love some help testing these speed changes on other devices.
Before attempting to merge the changes made in the new v1.0.51 is was getting a 3-5x speed increase in hashing and fingerprinting on my iPod Classic 7th Gen with an iFlashQuad (1x512GB MicroSD).

This could be easily improved further by automatically flagging iPod nano and shuffles as SSD based, or even running a small disk speedtest on the later gen iPod classics to try automatically determine this.

More details will be added as this pull request is closer to completion ❤

[B0N3head]

Also small question, why is there an inconsistency in directories between the logs/settings/bin files?
I chose to store the device_tags.json along side the base settings file, but why is there a split like this?

Current Setup

Name	Default Dir
Settings	%APPDATA%\iOpenPod
Logs	%USERPROFILE%\iOpenPod\backups
Required binaries	%APPDATA%\iOpenPod\bin (where fpcalc.exe is stored)
Backups	%USERPROFILE%\iOpenPod\backups
Cache	%USERPROFILE%\iOpenPod\cache

I was close of pushing a change of making all setting/log files to be stored in the %appdata% dir and the main files (fpcalc.exe/backups/cache) to be stored in this %userprofile% dir, but as unsure if it goes against a design decision or some oddity with the macos/linux versions.

Proposed change

Name	Default Dir
Settings	%APPDATA%\iOpenPod\settings.json
Logs	%APPDATA%\iOpenPod\logs\iopenpod_DDMMYY.log
New device tags	%APPDATA%\iOpenPod\device_tags.json
Required binaries	%USERPROFILE%\iOpenPod\bin
Backups	%USERPROFILE%\iOpenPod\backups
Cache	%USERPROFILE%\iOpenPod\cache

Thoughts???

[TheRealSavi]

i'd recommend storing device tags within the iopenpod json file on device. a seperate file is fine too if you don't want to touch that giant file. on device file storage is correct for this case as it will maintain the tags between computers for multi pc users.

[TheRealSavi]

as for the current setup for those files, i wouldn't touch them. moving them would mean we'd have to add code to check both spots so current users don't lose their configuration, and i just don't think it's worth the hassle.

[B0N3head]

i'd recommend storing device tags within the iopenpod json file on device. a seperate file is fine too if you don't want to touch that giant file. on device file storage is correct for this case as it will maintain the tags between computers for multi pc users.

Yeah ok easy, I'll give a shot at writing to the iOpenPod json file on the device itself (and ofc checking backwards comparability). I didn't even think about persistent settings across pcs.
I only have the single iPod at the moment for testing. I was considering adding this as an experimental feature as the 4 extra workers help hugely with the 7th gen but I'm not sure how much they would on a nano or shuffle. Worst case the old chips used on the nano and shuffle don't have the same read write speeds as expected and the extra workers would overflow the IO and make it slower.

Minor update:

The automatic disk based speedtests had too much variance between runs for them to be a concrete answer of SSD vs HDD (plus stressing an old HDD or cheap HDD to flash probably isn't the best idea). So for the current build I'm just prompting on newly (non tagged) devices if it is SSD or HDD based.

Currently under a large workload, so I most likely won't be working on this till the 25th, I'm sorry :(

Love this project 🙏🙏