second time flashing

[amlwwalker]

Hey
There is an issue with flashing the arduino after the first time.
The issue resides in the fact that each time you flash the arduino the device ID for the serial output buffer increases, but not for the input buffer (I think).
Have you gotten over this? For me everything is great the first flash, but after that I have to reset the plugin to flash the arduino again.
P.S Could you reference your reading material for learning how STK500 works please, I would love to read it

[mrjones]

Yup, I just tried it out and it looks like you're right about the bug.

No promises, but I'll see if I can take a real look this week (The STK code is such a mess, I bet it's something dumb).

In the meantime: I never found a great, self-contained resource for learning STK500. I used a mix of uploading programs with avrdude in very-very-very verbose mode (i.e. -v -v -v -v), and some official Atmel docs. It wasn't always pretty :)

I found some of my old notes here, if you want to check them out (I think I just made them public ... but let me know if you can't access them):
https://docs.google.com/document/d/1zHmnhyH-FsLMJEfgUmBkguRs-uSWo9oIZ3PNt9ltTWo/edit
It's not very well organized, but does have some links to good material, as well as a partially annotated analysis of one instance of an upload.

[mrjones]

The STK500 code was a real mess, so I moved it to stk500.js, and tried to give it a little bit of structure in the process. I left the old implementation in uploader.js for now, but I'll probably remove it soon assuming the new code works ok.

The new code can be accessed via the "STK500 (beta)" option in the dropdown, and seems to be working for multiple consecutive uploads for me on my arduino uno.

The code still is NOT very pretty, but now at least there's some structure, so there's a glimmer of hope of cleaning it up some ... so maybe I'll do that next.

Anyway, let me know if things are working better for you.

[amlwwalker]

Hi again, sorry its taken me a while to get back to you on this.
I have just been testing your code, and it seems now the same thing happens each time which is great - when I use STK500 beta.
Kudos by the way on working out how the STK500 is functioning, that must have been a nasty task I am going to take a look at your google doc - yes btw, it is public.

The bad news:
I have a web server hosting hex files - I generated the hex files in the Arduino IDE in verbose mode, then copied the .hex to the location the server is running from (that way I know the .hex works on UNO). take a blink example or something....

Whats strange is it does seem to flash or atleast communicate, however what occurs on the arduino is very strange - I get a continual three fast flashes, and then a long pause repeating continually. It looks to repetetive for me to think that its not some sample application you have written and it is grabbing it from there, rather than from my server. However I cant find in your code anywhere a hardcoded address of your server serving that .hex file - so its making me wonder if I'm going mad.

I've set the URL to my url (http:/127.0.0.1:9000/hex) in the UI, and I have stuck some console logs in it to make sure its reading my file, which I think it is, but still its always flashing it with the same hex I mentioned above - as far as I can tell....
My URL outputs plain text back - I based it on your end point that the sketch URL defaults to (sample):

:100000000C9435000C945D000C945D000C945D0024
:100010000C945D000C945D000C945D000C945D00EC
:100020000C945D000C945D000C945D000C945D00DC
:100030000C945D000C945D000C945D000C945D00CC
...
...
...
:1007F00020F0A21BB30BE40BF50B661F771F881FBD
:10080000991F1A9469F760957095809590959B0152
:10081000AC01BD01CF010895EE0FFF1F0590F491CB
:08082000E02D0994F894FFCFCC
:10082800686F7764790000000000CC0262036202FE
:0608380093027302BC02F2
:00000001FF

To give you an idea of the output on the console, I am getting:

-- STK500::WritePageData
logging.js:41 STK500::Writing [100,0,128,70,224,45,173,1,9,149] -> 41
logging.js:41 WRITE: {"bytesSent":133}
uploader.js:42 READ: {"connectionId":41,"data":{}}
logging.js:41 STK500::HandleRead: [20]
logging.js:41 Buffered 1 new bytes. Total is now 1, and waiting for 2
uploader.js:42 READ: {"connectionId":41,"data":{}}
logging.js:41 STK500::HandleRead: [16]
logging.js:41 Buffered 1 new bytes. Total is now 2, and waiting for 2
logging.js:41 ==== STK500::WritePage: 16
logging.js:41 -- STK500::LoadAddress 16
logging.js:41 STK500::Writing [85,8,0,32] -> 41
logging.js:41 WRITE: {"bytesSent":4}
uploader.js:42 READ: {"connectionId":41,"data":{}}
logging.js:41 STK500::HandleRead: [20,16]
logging.js:41 Buffered 2 new bytes. Total is now 2, and waiting for 2
logging.js:41 -- STK500::WritePageData
logging.js:41 STK500::Writing [100,0,128,70,153,31,26,148,105,247] -> 41
logging.js:41 WRITE: {"bytesSent":133}
uploader.js:42 READ: {"connectionId":41,"data":{}}
logging.js:41 STK500::HandleRead: [20]
logging.js:41 Buffered 1 new bytes. Total is now 1, and waiting for 2
uploader.js:42 READ: {"connectionId":41,"data":{}}
logging.js:41 STK500::HandleRead: [16]
logging.js:41 Buffered 1 new bytes. Total is now 2, and waiting for 2
logging.js:41 STK programming status: OK

Is that the expected output for a successful write?

FYI if the arduino is writing to TX it outputs a log message - something like "received data when not exepected" - not related to my issue, but thought I would mention it.

EDIT:

Right I just saw your .ino file and it looks exactly like the led output I am getting on the arduino, which suggests somehow it is getting your hex data rather than mine - although I have no idea where its coming from. Does that hexdata array need clearing or anything?

[mrjones]

Well this is interesting! The triple-blink is definitely the example program (which also lives at http://linode.mrjon.es/blink.hex).

I played around myself a bit by doing:

• Plugged in my Arduino Micro (and AVR109 board)
• If I change the URL to http://linode.mrjon.es/blink-micro.hex, I can successfully flash my Micro and get the blinking pattern
• If I keep using the http://linode.mrjon.es/blink.hex URL, it tries to upload different data that the Micro won't interpret (no blinking)

So, it at least can work, although I'm not at all surprised to find there's still a bug.

---

Looking at the logs, it appears we're trying to program page #16, which I think must be the final page (because it prints "STK programming status: OK" at the end. We reverse engineer the final commands:

"STK500::Writing [85,8,0,32] -> 41"
First byte: 85 is 0x55 which is the LOAD_ADDRESS command
Next two bytes are the address, 0x0800 or 2048, which indeed looks like page #16 (128 byte pages)
32 is CRC_EOP to say we're done the command.

"STK500::Writing [100,0,128,70,153,31,26,148,105,247]"
First byte: 100 is 0x64, which is the PROGRAM_PAGE command
Next two bytes: 0, 128 says that we are going to program 128 bytes worth of data
Third byte, 70 is 0x46, which tells stk500 we want to program FLASH_MEMORY
So the data starts at byte 4: 153, 31, 26, 148, 105, 247 (and then it gets clipped), which would be hex 99 1f 1a 94 69 f7

Long story short, we're trying to write "991f1a9469f7...." at offset 2048. I don't see that data at all in http://linode.mrjon.es/blink.hex , but I do see it in your snippett (in the right place):

:10080000991F1A9469F760957095809590959B0152

I forget all the details of the format of the hexfile, but it's basically:
[:][10][0800][00][991F1A9469F760957095809590959B0152]
So it's trying to write 99F1A969F7 to 0800 (which matches right up with the logs).

So, it's at least trying to program the correct data.

This seems to leave us with a few possibilities:

1. It's successfully programming the data in your snippet, and that hex file is causing it to do the triple blink. Maybe it's worth verifying that you're webserver has the proper hex file?
2. It thinks its programming the data in your snippet, but it's actually failing silently. So, maybe somehow you got the blink data uploaded successfully once in the past, but when you try to write something new you're just over-writing it. Maybe try re-flashing your board to some other program with the Arduino IDE (or something), and see what happens (i.e. if you're able to overwrite that program, or if stays flashed).
3. "Does that hexdata array need clearing or anything?" Something along these lines seems quite plausible too. I don't see it yet, but I'll keep looking.

---

One FYI: I'm moving development into a new subdirectory "extension2".

The old build/test framework was so naive and bad and I just couldn't deal with it anymore. It turned out a bunch of tests had syntax errors in them, and weren't compiling ... however it was still reporting them as PASS-ing. So, I've got a new framework that seems much better, but it involved making moderate changes so I moved it all to that extension2 directory. I left the old directory in place so as to not break it in the meantime, eventually I'll delete it but no rush ... however, in the meantime all new work will probably be inside extension2.

[mrjones]

Ok for what it's worth, I'm not getting similar behavior to you: It thinks it's uploading the right bytes, but the program isn't changing. (It seems to work fine with avr109, so it must be a bug in stk500.js)

I'm not sure I'll get to fixing this tonight or not, but hopefully now that I can reproduce it, then I can fix it sometime in the next few days.

[amlwwalker]

Hi again, - just added you on Twitter :)
So you say you are not getting the same behaviour as me, yet you think the program isnt changing? Sorry I am just wondering whether you made a typo in that last paragraph there?
Im going to continue to have a look, and see where it could be getting the hex from, because even if there is a bug in stk500.js, it must be pulling the data from somewhere - even on the first run so I am going to try and remove all links to your server before running it and see if I can get mine to be the data.....

FYI in serialmonitor.html you are looking for module.js in extension2 - it doesnt exist as far as i can tell....

[mrjones]

I was able to reproduce the issue, there were actually two bugs in stk500.js which were errors from porting the old uploader.js code. It was writing the right data, but it was writing it to the wrong addresses. So, the program wasn't ending up where it needed to, and we were left with some mis-mash of the new program and whatever was on the board previously. I've fixed both bugs, and things seem to be working better for me!

There's still the issue where it prints "no read handler for..." once you run it more than once, which I need to fix. However, that doesn't seem to be interfering with the uploader.

---

module.js is generated by a script now. Unfortunately, it takes some new tools to generate it properly:

Motivation: The new test framework I'm using (http://mochajs.org/) uses node-js style "require" and "exports" semantics to define dependencies between modules. Of course, chrome doesn't support "require".

So we need to use Browserify (http://browserify.org/) to transform the files in "lib" into something that Chrome will accept. I decided to use Grunt (http://gruntjs.com/) to manage the browserification process [1]. The Gruntfile.js in extension2 means if you just type "grunt" in the extension2 directory it will create the module.js file.

( [1] I'm realizing as I type this that grunt is probably overkill, since it should be possible to do what it does with one command. This generates the exact same file, without grunt:

browserify lib/*.js -o extension/js/module.js

)

So, clearly this is a bit of a work in progress. Sorry you have to see all the warts, but the mocha test framework is so many miles ahead of what I had before that I think it's worth it.

[amlwwalker]

Hi MrJones!
Excellent! I will pull everything now and have a go - before coming across your work, I did find code that flashed the arduino using Node, however, as you say Chrome doesnt support that so I tried browserify - I ended up with an 8000 line file as the output with a bug in so I am a little scarred by browserify - but I will give it a go and see if I can get it to do what I need.

Out of interest, Browserify is just required to generate the plugin right?
So I understand:

pull master
go into extension2(?)
from inside extension2 run your browserify command
install plugin
Ok lets try this!

FYI I am pleased you are thinking out loud, this is a great bit of work that I would love to use and understand and be part of.

---

Excellent, I had to use grunt, I am not sure I understood the browserify command - specifically the lib/* part - that dir doesnt exist, but I might be being an idiot....

I am going to have a play and have a look around. This is great dude!

[amlwwalker]

Hey again.

Most things seem to work. I am looking at the serial output at the moment - i.e displaying the serial.println commands in the plugin. It seems to work fine except when you disconnect, wait a moment and reconnect - you get a whole load of output - as though a buffer is still being filled while its disconnected and then everything is dumped back - it messes up the formatting.
Can you suggest a way to clear the buffer on reconnect so that this doesnt happen?

Thanks again,

Alex

[mrjones]

Sorry for the delay, got distracted with other things.

The extension used to add a new listener callback every time it connected. And it never never removed the callback. Which means that on the Nth connection, there were N listeners all listening and then racing to print out the same message. For me, this led to lots of repeated junk in the console.

I just committed a change (7f9af28) which removes the listener at disconnection so that we never have more than one listener going at a time. I'm not totally sure if this will solve your problem, but I think it might.

Have a look and let me know?