Workflow for avr build

[brentfpage]

(Following up on some of the discussion in #381.)
This PR:

• adds a workflow (firmware.yml) for the firmware
• makes edits to continuous.yml:
  • adds a firmware compilation job
  • adds support for the mac build scheme described in the bullet below
• revises mac.yml so that:
  • when mac.yml is called by continuous.yml, the firmware .hex asset produced by continuous.yml is packaged into the mac desktop interface
  • when mac.yml is called in some other context, the firmware .hex asset produced by the most recent call to continuous.yml is packaged into the desktop interface
• and similarly for the other platforms' workflows

The versioning is centralized, but not automatic. Namely, the current version of the firmware has to be set in .github/actions/define-firmware-ver-action/action.yml. At compile time, the firmware version defined there gets injected:

• into AVR_Code/.../src/main.c as the macro FIRMWARE_VERSION_ID, which previously was defined in AVR_Code/.../src/globals.h
• into the desktop interface as the macro EXPECTED_FIRMWARE_VERSION, which previously was defined in Desktop_Interface/genericusbdriver.h

Also, the inclusion of the version number in the .hex file names has been retained, so they are, e.g., labrafirm_0007_0{1 or 2}.hex. To me, this is worthwhile because then the firmware that's packaged into a given release can be very easily determined.

[brentfpage]

Copying from a previous draft of this PR:

---

Posted by @mi-hol

@brentfpage I like the approach documented above and would approve it. With the number of changes a real review seems impossible though.

Lets see how other maintainers react.

The is only a minor nit pick

Namely, the current version of the firmware has to be set in a repository-wide variable, AVR_VER

I'd suggest to change name of variable from AVR_VER to AVR_FIRMWARE_VERSION so its purpose is completely self explanatory

[mi-hol]

@brentfpage I triggered re-run of all workflows after merging #404 but they still fail

[brentfpage]

@brentfpage I triggered re-run of all workflows after merging #404 but they still fail

I just added AVR_VER as a repo-wide variable here (harmless and reversible) , which should make the mac build work. The other builds with automated firmware packaging have not been implemented. The firmware version is now defined in .github/actions/define-firmware-ver-action/action.yml

[brentfpage]

As an aside, the firmware that's currently configured to be packaged is that from the most recent "continuous release" in my fork. This practice will be necessary unless/until this PR is fully merged into the genuine repo and a continuous release is run that generates firmware files.

[mi-hol]

added AVR_VER as a repo-wide variable here (harmless and reversible) , which should make the mac build work.

Well I re-ran and it still failed :(
Is something preventing you from running these actions in your fork?
Running them would give you the confidence that the changes actually work as expected and is the key to merging them quickly.
I trust in contributors to provide 100% working changes after initial discussions have completed within the team about a proposed approach.

scoperangeenterdialog.cpp:4:131: warning: unused parameter 'delay' [-Wunused-parameter]
    4 | scopeRangeEnterDialog::scopeRangeEnterDialog(QWidget *parent, bool buttonVisible, double yTop, double yBot, double window, double delay) :
      |                                                                                                                                   ^
genericusbdriver.cpp:479:75: error: expected expression
  479 |     qDebug("EXPECTING FIRMWARE VERSION 0x%04hx", EXPECTED_FIRMWARE_VERSION);
      |                                                                           ^
genericusbdriver.cpp:484:45: error: expected expression
  484 |     if((firmver != EXPECTED_FIRMWARE_VERSION) || (variant != DEFINED_EXPECTED_VARIANT)){
      |                                             ^
1 warning generated.
2 errors generated.

The other builds with automated firmware packaging have not been implemented.

Please add these changes, test, change status to "ready for review" and we'll review again after that has happened.

[brentfpage]

Unfortunately I had missed this key point:
"[variables] are not passed to workflows that are triggered by a pull request from a fork"
from the heading of https://github.com/espotek-org/Labrador/settings/variables/actions . So, the workflow won't work in full unless/until the PR is merged. As a stop-gap solution, I simply hard-coded the value of AVR_VER (0x0007) in mac.yml. The mac build now works. Now obsolete (3/19/2026)

The PR is still a draft (only one workflow has been implemented), so I'll hold off on indicating that it's "Ready for review".

[EspoTek]

Just to add some hairiness, the Windows installer currently needs to be manually updated whenever we bump the firmware version.

This is one of the reasons we were sticking with pre-built binaries for everything.

I think it makes sense to have a job for automatically building the firmware, but unless we find a clean solution to the Windows Installer problem I don't think we want to be auto-pulling new builds in.

[brentfpage]

@EspoTek Is it possibly the case that the manually updated installer is downloaded in windows.yml with this line?

Invoke-WebRequest -Uri "http://espotek.com/ai167.msi" -OutFile "ai167.msi"

If so, could you please elaborate on what steps are necessary for generating that .msi file?

[EspoTek]

@EspoTek Is it possibly the case that the manually updated installer is downloaded in windows.yml with this line?

Invoke-WebRequest -Uri "http://espotek.com/ai167.msi" -OutFile "ai167.msi"

If so, could you please elaborate on what steps are necessary for generating that .msi file?

That like there is installed a (proprietary) program called "Advanced Installer", that's converts the .aip (Advanced Installer Project) file to the .msi package.

If we ever increment the firmware version, we'll need to either manually add them to the Advanced Installer project or possibly take advantage of some Advanced Installer feature that automatically adds them for us?

[mi-hol]

we'll need to either manually add them to the Advanced Installer project or possibly take advantage of some Advanced Installer feature that automatically adds them for us?

Updated
@brentfpage @EspoTek I started already some time ago a branch to simplify & standardize the Advanced Installer project.

• Therefore continuing to improve the currently used approaches seems to be a short-term fix only
• mid-term I'd prefer to simplify the whole handling of firmware variants and updates.

Reasoning is:

• major differences in code dealing with USB for unclear (aka undocumented) reasons (at least to me)
• many "loose ends" due to unmanaged dependencies to old USB library releases

The open questions in this regard are for me:

• can the current reasons for needing 2 firmware variants be eliminated?
  This came up because I'm unaware of other devices needing reflashes of firmware "on-the-fly" and a related error in LabraScope Currently implemented Firmware version check & message are incorrect! jdar0011/Labrador_ImGui#3
• can the process of firmware flashing be unified for Windows & other OSes?
  This came up because only Windows uses external programs to flash the firmware

Does my reasoning make sense?

[mi-hol]

@EspoTek Is it possibly the case that the manually updated installer is downloaded in windows.yml with this line?

Invoke-WebRequest -Uri "http://espotek.com/ai167.msi" -OutFile "ai167.msi"

If so, could you please elaborate on what steps are necessary for generating that .msi file?

That link there is installed a (proprietary) program called "Advanced Installer", that's converts the .aip (Advanced Installer Project) file to the .msi package.

If we ever increment the firmware version, we'll need to either manually add them to the Advanced Installer project or possibly take advantage of some Advanced Installer feature that automatically adds them for us?

To reduce dependency on the specific server http://espotek.com via insecure http protocol I'm in favour of converting this piece of code to a GH action from marketplace. I'd be happy to own that task.

[EspoTek]

• can the current reasons for needing 2 firmware variants be eliminated?
  Not simply - this is unfortunately a limitation of the ATXMEGA microcontroller we use. Long story short, we require 750kB/s of bandwidth and the ATXMEGA's USB hardware only lets us allocate these in blocks of 8, 16, 32, 64, 128, 256, 512 or 1023 kB/s. On Windows x64, we can get away with 6x 128kB/s endpoints because libusbK knows which endpoint it's communicating with. On other platforms we need to use 1x 1023kB/s endpoint because regular libusb abstracts this away.

It might be possible to switch between the two modes (1x 1023 byte endpoint vs 6x 128 byte) dynamically, this but we'd need to have a whole lot of dynamic code running on the MCU.

• can the process of firmware flashing be unified for Windows & other OSes?

This is probably more doable - we'd need to build dfu-programmer as a DLL. But honestly, I don't really see a major advantage from this, since the system we have in place already works well.

[mi-hol]

we'd need to build dfu-programmer as a DLL.

Why the Labrador project?
dfu-programmer has its own lifecycle in a separate project.
This is from my view an example for an "unmanaged dependency"

But honestly, I don't really see a major advantage from this, since the system we have in place already works well.

My experience tells me that at some point in time any code dependency will stop working and suddenly a nightmare surfaces because the skills to maintain it are not there.

From an external view many dependencies have been copied into the project to make it work but that is not a sustainable setup for the long-term unfortunately.

Reducing the amount of code required would be a strategy to survive.

[brentfpage]

I looked into utilizing the Advanced Installer Github Action for the windows workflow, but unfortunately the Action requires an enterprise license. So, it looks like sticking with the current approach is the way forward. As for

the Windows installer currently needs to be manually updated whenever we bump the firmware version

@EspoTek could this be as straightforward as manually updating the firmware file paths in Labrador.aip ? Currently in this PR, the paths in Labrador.aip have been changed to easily identifiable dummy paths that are replaced with real file paths by windows.yml. This is specifically done using a shell command, but in the future maybe it can instead be done using /UpdateFile. I wasn't able to get /UpdateFile to work for this most recent PR draft but may try again later.
On a related note, should the LABRA~[10 or 11] parts of the FileName entries in the aip be changed when the firmware is updated? In this PR, currently only the 000N part of labrafirm_000N_01.hex gets changed. Lastly, I deleted the labrafirm_000N_02.hex files from the aip, which simplifies the windows workflow and seems likely harmless.

[mi-hol]

updated:
@EspoTek @brentfpage I'm not convinced AIP to be the best way forward because:

• correction based on this comment we are already using CMake and version 4.3 introduced new capabilities for package building
• the currently used AIP version is outdated
• the license key was lost
• removal of 32bit version is an open action

that said short-term I'd suggest to just stick with

manually updating the firmware file paths in Labrador.aip

Then address above topics in a next step trying to standardize the package building with CMake

Would this approach make sense?

[turboencabulator]

This PR feels like it has scope creep. Can we get a single workflow to build the existing firmware now, then worry about firmware version updates and packaging issues separately in a future PR? I think the latter needs a lot of thought and discussion.

[turboencabulator]

we are already using CMake and version 4.3 introduced new capabilities for package building

As far as I'm aware, nothing in this project uses CMake. (And personally I'd like to keep it that way, every project I've seen that uses it is constantly tweaking things to get it right.)

[mi-hol]

As far as I'm aware, nothing in this project uses CMake.

@turboencabulator oops I confused Labrador with another project. Thanks for making me aware of my error :) l
I'l fix the statements made asap.

[brentfpage]

Can we get a single workflow to build the existing firmware now, then worry about firmware version updates and packaging issues separately in a future PR?

Good idea – the PR now just includes a new action for defining the firmware version and a new workflow for compiling the firmware. I ran the firmware workflow in my fork, uploaded the resulting variant 02 .hex asset to my board, and confirmed that the board then interfaced properly with the Labrador UI on my mac. I also made sure that the variant 01 .hex differs from the variant 02 .hex (as expected), but I don't have an x64 windows machine so can't confirm that the variant 01 .hex actually works. If someone has such a machine, could you please upload labrador_0007_01.hex from here to your board and test its functionality?

[mi-hol]

If someone has such a machine, could you please upload labrador_0007_01.hex from here to your board and test its functionality?

will do but will likely need until Wednesday

[EspoTek]

I looked into utilizing the Advanced Installer Github Action for the windows workflow, but unfortunately the Action requires an enterprise license. So, it looks like sticking with the current approach is the way forward. As for

the Windows installer currently needs to be manually updated whenever we bump the firmware version

@EspoTek could this be as straightforward as manually updating the firmware file paths in Labrador.aip ? Currently in this PR, the paths in Labrador.aip have been changed to easily identifiable dummy paths that are replaced with real file paths by windows.yml. This is specifically done using a shell command, but in the future maybe it can instead be done using /UpdateFile. I wasn't able to get /UpdateFile to work for this most recent PR draft but may try again later. On a related note, should the LABRA~[10 or 11] parts of the FileName entries in the aip be changed when the firmware is updated? In this PR, currently only the 000N part of labrafirm_000N_01.hex gets changed. Lastly, I deleted the labrafirm_000N_02.hex files from the aip, which simplifies the windows workflow and seems likely harmless.

Updating the .aip directly using sed or whatever the Windows equivalent is seems like a good approach to me. I've looked at it in Cursor and the filenames at least are plain text. I'd probably encode the full filename in the .aip rather than the "N" but that's just me (EDIT: maybe not, haha. Just read the code and it seems sensible as-is!).

Deleting the _02 will cause issues as it's used by the 32-bit build.

[EspoTek]

Also I'm happy to ship this is variant 01 works. Looks good!

[brentfpage]

One last thing I just did is to add the firmware workflow to continuous.yml. This will be convenient if/when a new PR is opened for packaging the firmware automatically into UI releases. Also, it's generally useful to have firmware versions tied to UI versions, which including the firmware in continuous.yml accomplishes. Maybe it would be even better to include the firmware in the continuous releases but somehow hide the firmware in the releases page, but there's no way to do that as far as I can tell.

[EspoTek]

Can't we just not call upload_artifact?

[brentfpage]

I think the .hex products would then never make it out of the firmware workflow's workspace, which gets deleted when the workflow exits. Even if an asset is uploaded using upload_artifact, it is only available for 90 days. There appears to be an additional upload step in continuous.yml that transfers all the assets (which have already had upload_artifact called on them) to some release directory, where they don't have an expiration date. Given this state of affairs, unless hex assets are included in the repository directly, I think that uploading them to the release directory is necessary.

[EspoTek]

Wait, why is this the case? We'll run this script every time we rebuild anyway, right?

[mi-hol]

x64 windows machine .., could you please upload labrador_0007_01.hex from here to your board and test its functionality?

Tested with Labrador version from commit 8fdaf80
and your firmware file.

no wires connected:

but with just wires connect it seems to pick up a lot of noise preventing exact measurement
need to crosscheck with original firmware though

disconnecting USB gave:

  winUsbDriver destructor ran!
  genericUsbDriver dectructor entering
  genericUsbDriver dectructor completed
  Reinitialising USB driver!
  Making USB Driver invisible!!
  pipeID 0 = 129
  pipeID 1 = 130
  pipeID 2 = 131
  pipeID 3 = 132
  pipeID 4 = 133
  pipeID 5 = 134
  Generic Usb Driver setup complete
  driver =  winUsbDriver(0x213ff398460)
  newGain = 8.000000
  gainMask = c0c
  Not connected.  Ignoring Control Request!!!
  Null handle error in usbSendControl
  numBuffers is not equal to 0, 1 or 2

[EspoTek]

@mi-hol does it show any issues when you pump through a "weird" sin wave, such as 791Hz? This will show any issues where the USB driver doesn't recombine the 6 endpoints in the correct order.

[mi-hol]

does it show any issues when you pump through a "weird" sin wave, such as 791Hz

Sorry, thats beyond my technical abilities.
Maybe with detailed step-by-step instructions and prerequisites I would be able.

[EspoTek]

does it show any issues when you pump through a "weird" sin wave, such as 791Hz

Sorry, thats beyond my technical abilities. Maybe with detailed step-by-step instructions and prerequisites I would be able.

When I do it, I'll:

• Connect Signal Gen CH1 AC to oscilloscope CH1 with a wire (see pinout https://github.com/espotek-org/Labrador/wiki/Pinout)
• Set the Signal Gen CH1 to a Sin wive at 791Hz and 2V
• Check to see if there are discontinuities in the waveform (ie, it's not a "smooth" sine - it suddenly jumps up and down randomly)

[brentfpage]

This PR is now intended to be merged after #440. It just includes a simple workflow that builds the firmware using the Makefile in #440 and the github marketplace action "AVR make". The workflow's functionality beyond that is limited to accepting a user-specified target variant when the workflow is initiated from the github actions page.

[mi-hol]

This PR is now intended to be merged after #440.

@brentfpage in case this PR is ready, please update it's status from currently "Draft" to "Ready"

[mi-hol]

• Set the Signal Gen CH1 to a Sin wive at 791Hz and 2V

@EspoTek the max voltage is only about 1.654V. Is that expected?

log:

PS C:\Users\mholzer> ."C:\Program Files\EspoTek\Labrador\0.1.3\Labrador.exe"
QObject::connect: No such slot bufferControl::signalGenIn(bool)
QObject::connect:  (sender name:   'signalGenGroup_CH2')
QObject::connect:  (receiver name: 'bufferDisplay')
QObject::connect: No such slot isoDriver::setWindow(int)
QObject::connect:  (sender name:   'timeBaseSlider')
QObject::connect:  (receiver name: 'controller_iso')
opening "C:/Program Files/EspoTek/Labrador/0.1.3/waveforms/_list.wfl"
newName =  "Sin"
opening "C:/Program Files/EspoTek/Labrador/0.1.3/waveforms/Sin.tlw"
Length =  128
Divisibility =  4
opening "C:/Program Files/EspoTek/Labrador/0.1.3/waveforms/_list.wfl"
newName =  "Sin"
opening "C:/Program Files/EspoTek/Labrador/0.1.3/waveforms/Sin.tlw"
Length =  128
Divisibility =  4
Making USB Driver invisible!!
pipeID 0 = 129
pipeID 1 = 130
pipeID 2 = 131
pipeID 3 = 132
pipeID 4 = 133
pipeID 5 = 134
Generic Usb Driver setup complete
driver =  winUsbDriver(0x1c97ae63550)
axes =  QCustomPlot(0x1c96e677b30, name="scopeAxes")
bool __cdecl QCustomPlot::addItem(class QCPAbstractItem *) item either already in list or not created with this QCustomPlot as parent: 1964652949120
bool __cdecl QCustomPlot::addItem(class QCPAbstractItem *) item either already in list or not created with this QCustomPlot as parent: 1964652949120
scopeIn_CH2 true
Changed to mode 2
QObject::connect: No such slot MainWindow::enableLabradorDebugging()
QObject::connect:  (receiver name: 'MainWindow')
CHECKING CONNECTION!
Device opened successfully!
2 BYTES TRANSFERRED
BOARD IS RUNNING FIRMWARE VERSION 0x0007
EXPECTING FIRMWARE VERSION 0x0007
1 BYTES TRANSFERRED
FIRMWARE VARIANT = 0x02
EXPECTED VARIANT = 0x01
Unexpected Firmware!!








FIRMWARE MISMATCH!!!!  FLASHING....







deviceConnectedDisplay::flashingFirmware
UsbK_ControlTransfer failed with error code 31
FLASHING  "C:/Program Files/EspoTek/Labrador/0.1.3/firmware/labrafirm_0007_01.hex"
stdio_stage1 ""
sterr_stage1 "dfu-programmer: no device present.\r\n"
EXIT_CODE stage1 3
stdio_stage1 ""
sterr_stage1 "dfu-programmer: no device present.\r\n"
EXIT_CODE stage1 3
stdio_stage1 ""
sterr_stage1 "Erasing flash...  Success\r\nChecking memory from 0x0 to 0x7FFF...  Empty.\r\n"
EXIT_CODE stage1 0
stdio_stage2 ""
sterr_stage2 "Checking memory from 0x0 to 0x2FFF...  Empty.\r\n0%                            100%  Programming 0x3000 bytes...\r\n[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>]  Success\r\n0%                            100%  Reading 0x8000 bytes...\r\n[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>]  Success\r\nValidating...  Success\r\n0x3000 bytes written into 0x8000 bytes memory (37.50%).\r\n"
EXIT_CODE stage2 0
stdio_stage3 ""
sterr_stage3 ""
EXIT_CODE stage3 0
stdio_stage4 ""
sterr_stage4 "dfu-programmer: no device present.\r\n"
EXIT_CODE stage4 3
stdio_stage4 ""
sterr_stage4 ""
EXIT_CODE stage4 0
flashRet: 0
CHECKING CONNECTION!
Device opened successfully!
2 BYTES TRANSFERRED
BOARD IS RUNNING FIRMWARE VERSION 0x0007
EXPECTING FIRMWARE VERSION 0x0007
1 BYTES TRANSFERRED
FIRMWARE VARIANT = 0x01
EXPECTED VARIANT = 0x01
Connecting now!
0 BYTES TRANSFERRED
newDig
0 BYTES TRANSFERRED
Setup successful!
UsbK_ControlTransfer failed with error code 31
PSU DUTY CYCLE of dutyTemp = 16 OUT OF RANGE (could underflow on SOF)!!!  ABORTING!!!
UsbK_ControlTransfer failed with error code 170
ReinitUsb entering stage 2


winUsbDriver destructor ran!
genericUsbDriver dectructor entering
genericUsbDriver dectructor completed
Reinitialising USB driver!
Making USB Driver invisible!!
pipeID 0 = 129
pipeID 1 = 130
pipeID 2 = 131
pipeID 3 = 132
pipeID 4 = 133
pipeID 5 = 134
Generic Usb Driver setup complete
driver =  winUsbDriver(0x1c97ae63550)
ReinitUsbStage2 is returning
ReinitUsb Stage 1 complete
deviceConnectedDisplay::connectedStatusChanged running!
CHECKING CONNECTION!
Device opened successfully!
2 BYTES TRANSFERRED
BOARD IS RUNNING FIRMWARE VERSION 0x0007
EXPECTING FIRMWARE VERSION 0x0007
1 BYTES TRANSFERRED
FIRMWARE VARIANT = 0x01
EXPECTED VARIANT = 0x01
Connecting now!
deviceConnectedDisplay::connectedStatusChanged running!
0 BYTES TRANSFERRED
newDig
0 BYTES TRANSFERRED
Setup successful!
New voltage = 4.5
Going to send value  31
newDig
0 BYTES TRANSFERRED
0 BYTES TRANSFERRED
Trigger Level:  -4
Trigger sensitivity: 1
Trigger Level:  -4
Trigger sensitivity: 1
Trigger Level:  -4
Trigger sensitivity: 1
Trigger Type:  0
Trigger Type:  0
Trigger Type:  0
Changed to mode 2
0 BYTES TRANSFERRED
128 BYTES TRANSFERRED
0 BYTES TRANSFERRED
128 BYTES TRANSFERRED
newGain = 8.000000
gainMask = c0c
0 BYTES TRANSFERRED
0 BYTES TRANSFERRED
Multiplying by  0.0625
Buffer shifted by -4
Buffer shifted by -4

[brentfpage]

I wonder why

FIRMWARE VARIANT = 0x02
EXPECTED VARIANT = 0x01
Unexpected Firmware!!

appears in the log above. I just uploaded labrafirm_0007_01.hex from the link I sent above to my board and then read the board firmware variant and got 01, as should be the case.

[mi-hol]

I wonder why

@brentfpage this correct because I share the board with Windows and Android clients. Nothing to worry about!

[brentfpage]

I wonder why

@brentfpage this correct because I share the board with Windows and Android clients. Nothing to worry about!

Ah OK got it. One thing to look out for is that the app will flash the firmware .hex file that got packaged in with the app when the app was built. This .hex file is different from this PR's .hex file unless you have gone to the trouble of modifying the app source files and rebuilding the app. Maybe I'm saying stuff that isn't relevant to what you're focusing on at the moment but I figure it's best to mention.

[mi-hol]

This .hex file is different from this PR's .hex file

Thanks, I thought to have considered that, but when I checked the new firmware was actually NOT used for the test above.

Changed my setup to:

And re-ran the test but now got USB connection issues on Windows and Android.
Neither OS detects the divie as connected but the red LED on board is on.

From looking at the log I'm unable to tell what commit of Labrador or firmware I used.
1.) I'd suggest to add both to the logs.

2.) For the release it seems to be advisable to change "labrafirm_0007" to "labrafirm_0008" or how would you envisage to get the new firmware deployed?

log:

NewFirmware.log

[brentfpage]

Neither OS detects the divie as connected

That's obviously concerning. I don't think there's a problem with the new labrafirm_0007_01.hex of a severity that would cause this issue. I was able to upload it to my board, have my mac read that the variant is incompatible with the mac, and then have my mac overwrite the 01 variant with the 02 variant. This process obviously requires robust connectivity between the mac and the board.

From looking at the log I'm unable to tell what commit of Labrador or firmware I used.
1.) I'd suggest to add both to the logs.

Yeah I agree this should be done

2.) For the release it seems to be advisable to change "labrafirm_0007" to "labrafirm_0008" or how would you envisage to get the new firmware deployed?

Yeah I think bumping the firmware version number would definitely be good for the release, even though the intent is that the new .hex files are functionally indistinguishable from the current .hex files.