SANE-less scanner integration

[activexray]

Hey! So following up on the other discussion of scanner support in NegPy:

I've been working on my own reverse-engineering of the SCSI driver for the Coolscan 9000 I have. I'm sick of having to boot up my windows VM with a dedicated FireWire card to scan just to bring them back into linux to invert. I know I could use Nikon's conversion but I used to use NLP and now I've been trying NegPy to get away from windows + lightroom.

I'm a decent way through the RE, I've got thumbnail scanning working and have a pretty big chunk of the API surface covered. My initial plan was to write my own standalone scanner program with GPU-accelerated renders for these massive medium format scans, exposing all the bells and whistles for the scanner that SANE does not support (it technically doesn't even support the 9000, only the 8000).

It would not be that hard to modularize what I do into a library that then you can call from somewhere else such as NegPy. I see there is already ScannerService, so I guess I would implement that? Once I flesh out all the details I'm sure there will be more discussion over how to expose features.

[marcinz606]

@rohanpandula more coolscan support :)

[marcinz606]

Rohan is working on something similar using direct-USB acquisition that will be integrated into NegPy
https://github.com/rohanpandula/coolscanpy/tree/main

[rohanpandula]

oh nice, a 9000 :)

I just walked this road with the LS-5000: tried to land my RE'd driver inside negpy (#497), and marcin's call, which was right, was device code lives in its own package with a python-sane-shaped surface (open / options / scan -> array), negpy consumes it as a thin adapter like python-sane or gphoto2. that became coolscanpy (github.com/rohanpandula/coolscanpy).

honest pitch: build the 9000 in there. transports are isolated per device on purpose, the 5000 is just the first entry, and you'd inherit session/reservation logic, receipts, a hardware-free test harness and the pypi plumbing instead of rebuilding them. clean ownership split: you own the 9000 transport/module (I can't even plug firewire in, I won't pretend to maintain it), I own the core and the 5000, and negpy gets one adapter for every coolscan.

command grammar should be mostly shared even though the transport differs: frame addressing, exposure vectors, multisample arming chains, IR windows, load/eject. most of that is decoded for the 5000 from wire captures, happy to share notes, might save you real hours.

worth watching #556 too, the session/handover seam there (device ownership for a whole batch, one release at the end) is what any external driver needs to coexist with negpy. and your standalone GPU app fits fine on top, lib core + your app is the same split I ended up with.

PS we are neighbors, I live in HP! If you ever want to shoot together shoot me a message :)

[activexray]

Few questions, and maybe we can take discussion elsewhere about it is out of scope here:

1. What is your driver reverse-engineered from? A quick read seems like a reimplementation of parts of SANE driver but some stuff is still delegated to SANE? I've been intentionally not looking at the SANE driver and just relying on a Ghidra decompilation that I've been working on with an LLM + a proxy DLL loaded in NikonScan to capture scsi traces.
2. I'm having a hard time following the data flow through your library. If I were to PR in 9000 support, how would that even go? There's a lot of 5000-specific code baked in there.

To be honest, I don't have a ton of interest writing any python. My intention is to write my library in rust as I've found it the best choice for working with hardware. This low level stuff really benefits from strong typing. Plus, I don't really have the patience for numpy to process multi-GB files compared to a compiled language. My idea was I was going to focus on my own library, write a GUI on top of that, and expose the library to python with py03 to be used here (or elsewhere).

By the way, you don't need an llm to write your prose :)

[marcinz606]

@activexray I'm currently making some changes in scanner implementation making generic ScannerBackend protocol. Currently i obviously only have SaneBackend talking to it but I want to make it easy to have for example your python-wrapped rust lib or what Rohan is doing pulled as dependency and just plugged in there. I'll probably have it merged to main soon.

[marcinz606]

I merged my changes to main. SANE-based coolscan support is a bit better + the ScannerBackend is more friendly towards non-sane integrations

[davide-b]

Hi all! I am very glad to see this move towards non-SANE implementation because I would love for NegPy to natively scan on Windows as well.

To this end, I would like to jump on the reverse engineering bandwagon. I own a Reflecta (Pacific Image) Proscan 10T and I was planning on writing a python bridge shamelessly porting the SANE code. That's why I'm grateful for the work on generic scanner interfacing that Marcin is doing.

Will post updates as soon as I have something!

[activexray]

I'll be working on my implementation here for the time being if anyone wants to keep track. I have a private repo with the RE work (it contains a lot of Nikon proprietary binaries so I don't think I can publish it). This public one doesn't do anything yet as I slowly learn how SCSI works and write something I trust. The slopped-together thing in the private repo is really quite bad but does do more. I'm still puzzling over how to reconstruct the image data from the bytes I get back, I can't make a ton of sense of it.

[activexray]

It took an unreasonable amount of time and I haven't seen it documented anywhere - but here is how you decode the output data stream from the coolscan 9000

https://github.com/activexray/nkscan/blob/main/src/scanners/ls9000ed/decode.rs

it was...non trivial. You need to know the secret that at 4000-dpi, the three sensor lines are 12 dots apart.

[activexray]

I don't know why the other coolscans don't have this logic? But in any case this streaming descrambler is blisteringly fast.

[marcinz606]

i forked you repo and started looking around if maybe i could contribute LS-50 to it

[activexray]

you might want to wait a few days while i get things architected a bit more, although i think i have it organized in a pretty reasonable way

[davide-b]

A little update on my Reflecta work: I have a working standalone python script that runs on Windows. I started integrating the command flow into NegPy by making a new backend called pieusb_backend.py, mirroring the sane_backend.py file. Right now, I hard-coded the NegPy interface to use my backend instead of the SANE and to bypass the Windows check, but I think a system to select a backend should be put in place, where I can slot my backend in, maybe with some automatic detection. @marcinz606 any thoughts about this?

Additionally, since @activexray is, like me, also working with a SCSI device, perhaps it would be a good idea to extract this to its own module. Though, despite both being SCSI devices, my scanner is USB while the coolscan is Firewire. Unsure what the best strategy is here. My current USB SCSI implementation is rather lean and has no dependencies other than pyusb, so it would be nice to keep it that way.

[marcinz606]

i think ideally it should detect valid backends based on device and give user the option to toggle by dropdown, maybe at the top of the scanning section.

[marcinz606]

@davide-b if it's a lot of device-specific code it would be preferable to have it as separate python package installed as dependency rather than putting it directly in negpy codebase

[davide-b]

You mean something similar to coolscanpy by @rohanpandula ?

[marcinz606]

yes, exactly. then device-specific code can be maintained separately from NegPy

[activexray]

RE transport: those SCSI-2 standard scanner CDB payloads are on the surface transport-agnostic. so my driver has their definition separate from the transport so i can stuff the same payloads into USB or raw generic SCSI (although i haven't implemented that yet nor do I have a scanner to test it on. I have a primefilm xas but i don't know what protocol it uses yet as im spending my energy on the coolscan.